 Hi everyone, good evening. Thank you all for coming to our session. We have a panel to present this evening on promoting open science and transparency across sub-disciplines of psychology and so we'll be featuring several editors from APA journals as well as our open science and methodology chair. So first I'd like to share a quick overview of open science and how it factors in the APA, the American Psychological Association. Open science improves the transparency, accessibility, and reproducibility of science and here I'm presenting in three different ways to ways that you can engage in open science practices. So for simplicity to provide an overview, you can look at how open science aids in confirmation of hypothesis, so things like registering your research, so being upfront about what you plan to do with your research study before you've done the research. There's open science practices that can aid with collaboration, things like data sharing so that others can do secondary data analysis on the data that you've collected or pre-print so that folks can see the findings of your study before it's been published in a journal. And then open science can also aid in transparency and so an example here is constraints on generality statements which allows you to be very transparent about how far your your research findings extend. Some open science practices that APA has engaged in over the last several years include things like open science badges and so here I've displayed the open data badge but one thing that was very important since a lot of the research we publish reports on human interaction was that we have the open data protected access badge and so that's something the APA was very active in advocating for and so that that allows you to state that you'll share your data but it's it's um there's protected access to it so you have to have that data requested from you so it's not completely open in the world. Another way that the APA has engaged in open science is through journal article reporting standards and so these are for quantitative qualitative and mixed methods help um authors really think about being extremely transparent and what the reporting in the research study. We have a APA journal branded data repository through open science framework so if you publish an article with us and you deposit you can deposit your data in that repository. We recently last year launched a joint psychological societies pre-registration of quantitative research and psychology so this is a template Fred Oswald was part of this as well. It was a group with British Psych Society and German Psych Society as well as Center for Open Science and ZEDPED and that was to create a template that that researchers could use to pre-register their research and then finally we created an open science and methodology committee a few years ago and as I mentioned Fred Oswald is our chair now and this committee is in place to consider open science particularly as it pertains to APA journals and make recommendations on how the journal should be thinking about open science. And so for example one change that came about because of the committee's recommendation was for the APA core journals to sign on the top guidelines and so that's been happening throughout this year. So today we'd like to specifically feature the ways in which different journal editors and our open science and methodology committee are engaging in open science practices and specifically we try to pick a subsection of journals to really highlight how the different disciplines engage sub disciplines in psychology engage in open science and so Dr. Ebe will talk about journal applied psychology and her work there. Dr. Leitch will talk about the interpersonal relations and group processes section of journal of personality and social psychology we have a recording from Dr. Yates who couldn't join us tonight but he's the editor of neuropsychology and then Fred Oswald will speak more broadly about open science practices and then we'll have a Q&A session so we hope that several of you will be able to to participate in that and so I will move on to Dr. Ebe now. Thanks Rose I appreciate that so I'm going to talk to you about our kind of foray into open science at the journal applied psychology so I'm the editor of the journal and have about have 15 associate editors that work with me next slide Rose. So just to give you a little bit of context about the journal applied psychology I pulled this off of our website we basically study the psychology work employment and so our journal focuses specifically on the cognitive behavioral and psychological processes at multiple levels of analysis that are related to the world of work. We focus on primarily research that's conducted in organizational settings although we also do publish experimental and simulation and some methodologically based work so our context is somewhat unique and that we're really studying people primarily in field-based settings. We do publish largely empirical research but we also do publish qualitative research which I think poses some unique issues for open science. Next slide Rose. So in terms of our policies related to open science I took over the journal in 2020 and at that time we knew that there was a at least some rumblings at APA about really signing on to the tops and also really promoting open science so early on several team members and I wrote a piece on methodological reporting and some developed some checklist we could use with the journal. This is not to replace APA standard jars for qualitative and quantitative but what we do find is that some of those checklists were kind of cumbersome and for the type of work we do we thought a more specific type of checklist would be helpful so I mentioned this because it was really a nice way to get our stakeholders thinking about open science in a way that they could really understand and so we didn't implement this as policy but we implemented it as best practice prior to the decision by APA to implement the top guidelines and be a signatory. So on our website we have pretty detailed information on submitting manuscripts and also open science and we actually spent a lot of time really crafting this message. Open science is not something that is that common to people and industrial organizational psychology and management and so we really wanted to make sure that our messaging was clear and concise and so several of my associate editors and I worked on a lot of language with the help of APA and really kind of tailored it to our unique audience and effective number first we will be implementing the top guidelines and you can see here just as an example the different levels at which we have decided to adopt APA I think very very wisely determined that the different editors could make decisions about top implementation implementation that worked for them and required at least a level one so on some of these we have level two we don't have replication list on this table because we've always encouraged replications at the journal but you can see here that we are getting ready to go live very soon so if you talk to me in a couple of months I'm sure I'll have some additional stories but we've done a lot of preparatory work and we feel really quite prepared. Next slide. Kind of collaterally with the movement to the top guidelines one thing that I did when taking over as editor is create kind of an ancillary website that's not so much associated with the journal and it's not something that is formally endorsed by APA but it was a way for us to really signal our values to authors to reviewers and really reach a broad audience and so this is just a screenshot of our of our website encourage you to look at it it's got a lot of great information for graduate students authors reviewers and we deliberately have a a sub page on open science and on one we have some really good people who are learning about open science we have a logical checklist also open science events that are happening we have links to APA and actually we also have links to what other journals in our set of psychology are doing and we've gotten really good feedback on this this website as a resource but particularly related to open science next slide and so I'll just I'll say a few things about let learn this was organizers asked us to talk about and one thing they particularly for our area sub area of psychology is that education is really important and luckily prior to APA's decision to implement the top guidelines there had been numerous sessions at division 14 talking about open science getting people to understand what it was about and and we realized that there's a lot of misconceptions people often think of it as kind of an either or like you have to share everything you have to pre-register everything so we've been really working hard in small ways to try to get people to understand that there's lots of ways to be open but also to really highlight the advantages but for us it's really been a culture change and it's a still a culture change in progress um another thing I would recommend is just starting small I mentioned that we started with our methodological checklist and some messaging around open science I do give quite a number of talks to universities about the journal and so I am inserting open science slides into those presentations to get people talking about it and I also think that considering kind of the unique issues facing your sub discipline is really important for for industrial organizational psychology in particular issues of data sharing can be can be fraught with problems and in many cases may not even be allowed we tend to class dietary data a unique issue with some unique ontological beliefs that in some ways run counter to some of the tenets of open science or at least the interpretation and so I think that poses some challenges I'd also I'd also encourage you to think carefully about the role of reviewers and associate editors we're a very high volume journal we had almost 1700 submissions last year alone and so the the thought of how to do this well without increasing tremendous burden on reviewers and associate editors is something that we've really given a lot of thought to we also have given a lot of thought to how to ensure that the system is fair for all authors and that involves for us making sure that reviewers are clear on their role as reviewers as it relates to open science to make sure that people are being treated fairly in the process and then the last thing is we did spend also a lot of time just really thinking about the nuts and bolts there's a lot of boxes to be checked in implementing open science particularly at at a journal and so we did spend a lot of time thinking about who's going to be checking which parts of manuscript submissions what is our peer review manager going to be doing what is the editor going to be doing what are associate editors going to be doing and also whatever reviewers are going to be doing and so I this will be helpful we implement mine in November 1st Rose Lillian I missed that last part but you're finished right yes okay good your audio held up very well until the end so we're good all right great I'm going to stop sharing my screen now because dr leach is next and he'll talk about journal social our journal of personality and social psychology um hello everyone rose can you just give me a thumbs up if my audio is okay great thank you um so yes I'll speak about from the perspective of the journal of personality and social psychology and as maybe some of you know social and personality psychology but especially social psychology has maybe a unique relationship to open science and that's partly because we were at the center of a number of highly publicized scandals in psychology and I think to have this discussion for social psychology in particular we have to first acknowledge um events about 10 years ago um that started the particular form of open science practices and social psychology and so that was um the discovery of of long-standing years-long research misconduct by Dietrich Staple in the Netherlands in 2011 um was a number of well-known publications um were discovered to be fabricated um and also what became clear in that process which was that in fact it was very very difficult um for any of the people who suspected that misconduct to actually expose it it took years um and so this this really was a was a was a lesson for us in um the lack of transparency in research practices um and then of course the other thing that happened which was a little closer to home at JPSP although as well before my time as editor there which was in 2011 in that same year um Darrell Berman colleagues published in our journal a paper with nine experiments um purporting to provide evidence of ESP and this of course um attracted a lot of criticism and um and uh there were editorials written and blogs and uh in many ways people um focused on this article by Darrell Berman um supporting using very traditional social psychological methods and nine experiments um to provide empirical evidence for something that many people um believe cannot be empirically validated and so this also led to a lot of hand-wringing and criticism and so those two cases really fixed the field of social psychology um on uh research practices and we've been pretty fixed on it for the last 10 years um and in the end I think it's led to a particular in some ways cultural shift toward open practices in social psychology but also you know quite a lot of um tension and discussion or what have you and many people even called the subsequent years after those two revelations in 2011 or those two controversies many people have called uh have said that the field is in crisis um and so a number of things have happened since and I think have been really productive and the first one is and I think this is relevant to to everyone no matter what sub-discipline or discipline you're in which is that our main society actually our main professional society the society for personality and social psychology convened a committee and distributed guidelines that were that were really um shared widely in 2014 and this was really a a kind of consensus statement um about the need for more open and transparent practices um and reporting in the field and I think in many ways it reinforced some of the messages of the journal article and reporting standards that that Rose mentioned a little while ago which was um produced by APA in 2008 but that a lot of social and personality psychologists hadn't really picked up um and I think it carried forward some of those messages and updated them to some degree and so therefore I think it's also really consistent then with the latest journal article and reporting standards published in 2018 and so now I think um at the journal we have the last few editors across the three sections have really been referring to the 2014 document by our professional society and also the 2018 journal article reporting standards to refer to what I think are are clearly consensual statements about best practice in the conduct of research and the reporting of research and then um from my team in my editorial 2020 and in every editorial letter we refer to those standards and we also refer to the publication manual by the American Psychological Association which is another very important and useful document for research practices and reporting and so in every communication that we have with authors and reviewers we try to reinforce this consensualization of standards of practices and reporting by referring to those keystone documents and I think this is a really important in many ways social and cultural dynamic of open science and that is um developing consensus and agreement whatever it is that we can agree on and consensualizing it and then all referring to some guiding documents or statements in in our communications and in our engagement and our evaluations of work and as as the previous speaker mentioned also getting reviewers and everyone involved in the process referring to that same standard set of standards basic minimal standards has been a major task for us at the journal okay so more specific policies we um all three sections of JPSP have endorsed the level two guidelines by top and then since January 2020 we were also required upon initial submission the completion of the open practices disclosure form which indicates where data would be made available where data would be made available and where materials are available and code the relevant code for programming or for analysis and so authors must explain why this is not possible or why it's not advisable so the standard is that people make data materials and code available and if there are reasons why this doesn't make sense then they really have to say so and so therefore now all published articles include in the author note reference to where the data materials and code can be found or some explanation in those rare cases where it's not accessible and so again now in every art in every issue people are also getting this message over and over again that this is the standard and it is something that all authors have to commit to upon initial submission and I think that's been an important change for us the other thing we've done specifically at the interpersonal relations and group processes section of the journal my personality and social psychology is um and I think this also goes to the previous speaker's point about the extra labor that goes into engaging open practices documented information and so for our team because I think of our unique relationship and social psychology to this it was important for us to have and so APA was very supportive of us having three methods and statistics associated editors who can scrutinize initial submissions scrutinize some of that additional evidence that's presented and in some cases reanalyze data or perform simulations of data and this is really empowered the entire editorial team to ask pertinent questions if things are unclear and to engage the material that's provided through open science in a way that's actually useful and productive in the initial editors assessment of the initial submission because otherwise the assessment of the material that's submitted through open science practices the materials the data would have you that actually relies on volunteer reviewers to make the effort and to have actually have the expertise to engage the material that open science is now presented to us and so we thought it was really important to get some technical support to our editors in those cases where it really matters but there is a question here of scale and of volume and how to manage this given uneven expertise and interest or motivation among reviewers and I think that's a really important question for us given the system of peer review that our journals use which is a voluntary system where is this labor and expertise going to come from so having lots of information and access to it doesn't serve anyone if there's no one to actually make use of the information so this is one step that we've tried to take the methods and statistics associated editors but it really is a stopgap measure it's not really a long-term solution to making the most use of the extra information that we have access to we do not require pre-registration of hypotheses but we have been encouraging it a voluntary pre-registration and we do as a matter of course encourage authors when they've been asked to conduct further studies or reanalyze data or what have you to engage in pre-registration at that stage so we've certainly been using that as part of the editorial decision making when people are conducting replications or extensions and there is a sort of a logic that's apparent to just make that early to make that commitment to actually pre-register replication studies or follow-up studies now because of social psychology's unique relationship to these issues I would say a good 60 to 70 percent of our submissions contain pre-registered studies actually and that number is going up every day but there too we have the question of quality of pre-registration not all pre-registrations are equal in their quality some are very vague some are very general and of course we have had some cases where reviewers or others have raised questions about deviations from pre-registration or the quality of pre-registration so that again that raises the issue of making use of the information that's provided okay so I just want to mention me but a couple things now about researchers and the broader involvement as I've already mentioned a key area of growth I think for us is engaging reviewers in the even application and understanding of open practices this has been a struggle so far even though we've tried to share what I think are consensual standards there's a lot of difference of opinion and at the moment I think it's been a little difficult because some reviewers are very committed to maybe what you might even call sort of level three top guidelines and I think those are the only acceptable ones and others to none at all and so this is introducing now another level of heterogeneity in reviews that editors have to manage and that authors have to contend with and so far I think the field at least doesn't have a good approach to dealing with that but at the moment voluntary pre-review is highlighting where there is a lack of consensus on what open science practices are expected of authors we've also really pushed and you require authors to commit to an a priori power analysis and also with an explicit rationale for expected effect sizes and this helps deal with not genuinely a priori power analysis because of course people can say that they've done an a priori power analysis but if they have no real rationale for the effect size that they expect and that a priori power analysis isn't really worth the numbers that it produces and so getting researchers to be reflective and conscientious about their practices and what design what number of participants what methods are most appropriate for reasonable expectations has been a really important thing for us to engage authors and reviewers on and I think we're making headway there but that again has been I think relying on us sharing information and educating all of those involved in the peer review process about the usefulness of power analysis and I will say I think that the COVID effect on slowing research which has been pretty dramatic in social psychology has actually given us an opportunity to more productively and constructively and less defensively engage authors because I think everyone's been slowed down and that slow down has I think led to authors and reviewers having a greater appreciation for the effort that goes into producing research and therefore in some ways it's kind of like the the slow food movement in many ways I think people have had an appreciation for the quality that can be gained by slowing things down and I think that that does open more people up to the gains of open science practices by sharing more by being more reflective and and all of us engaging in a little bit more careful assessment of what we've done why we've done it and and how convincing it is and essential part of that for us the last thing I'll mention is we've also been encouraging people to also engage in open and best practices of graphics and in fact the previous panel ended with a discussion of that and that's another thing that we really pushed that actually making data more transparent in graphical representation is another way to reinforce this idea that we all gain by sharing as much information as possible about what we've done and why and so I think I might be coming toward my end but that's what we've been doing at the Journal of Personality and Social Psychology to engage open practices and to try to to get all of those involved in a productive and constructive conversation. Thank you. Thank you Dr. Leitch. Our next presenter is Dr. Keith Yates from the Journal of Neuropsychology and I'll be showing his video presentation. My name is Keith Yates, I'm a professor at the University of Calgary and I'm really pleased to be part of this symposium on promoting open science and transparency across sub-disciplines in psychology and I want to thank the organizers and particularly Rose Sokolchang for inviting me to be part of this. Before I get started just by way of disclosure I want to acknowledge that I have current grants and research support from the Canadian Institutes of Health Research and Alberta Children's Hospital Foundation. I also get a small amount of book royalties or books that I've edited or authored and I receive an editorial stipend from the American Psychological Association. I'm sorry I can't join you in person, I'm speaking from my office in Calgary, Alberta, Canada. I find that many people around the world know where Toronto, Montreal, and Vancouver are but may not be familiar with Calgary and this is a picture of the lovely view across downtown Calgary to the beautiful Canadian Rockies and the reason that I'm not with you in person today is that I had the opportunity to take a weekend at the Issynavoid Provincial Park. This is a picture of that area. It's a lovely part of the Canadian Rockies and I'm really looking forward to spending the weekend there because it's a really lovely place with tremendous hiking and we're very lucky to have it in our backyard in the midst of this pandemic. Without further ado let me go ahead and move along. I'm going to start just really briefly by talking a bit about what open science is. We've already reviewed this previous to my talk but just to reiterate, open science promotes transparency, integrity, and reproducibility of science. In many cases this is done through the sharing of data with the establishment of repositories where investigators can share their data, their materials, their computer code, analytic code, and also the sharing of scientific plans which often translates into the pre-registration of studies including in many cases the pre-registration of data analytic plans. So how does this influence neuropsychology? For those of you who aren't familiar with neuropsychology, neuropsychology is the science of brain behavior relationships. It typically focuses on how the brain is related to human behavior and both healthy as well as brain disorder and population. So for neuropsychologists many of us focus on groups of individuals who have a brain disorder, say a brain tumor or a traumatic brain injury, or are affected by neurodevelopmental disorders such as learning disabilities or attention deficit, hyperactivity disorder. Although some neuropsychologists focus on healthy brains and what we can understand about behavior in healthy populations. So we've been heavily influenced by outside forces that have driven the open science movement, of course the Center for Open Science, but also funding agencies that have increasingly expected us as investigators and as journal editors to promote open science, the National Institutes of Health, Canadian Institutes of Health Research for example, have promoted and required increasingly that investigators deposit their data, make it publicly available, make their publications publicly available, and that's certainly had a significant influence in our discipline. Another area or another force that's been at work is medical journals. Many of the journals that we publish as neuropsychologists tend to focus on medical disorders or populations and a number of those journals have also pushed in the direction of open science, particularly when it comes to randomized controlled trials or systematic reviews but not limited to that. So what we've seen over the past decade has been the establishment of data repositories that allow investigators to share their data readily with like-minded investigators. Some of these are government initiated efforts such as FitBur, which is the data repository initiated by NIH for investigations focused on traumatic brain injury. Some of these are voluntary efforts to pool data and put them into available repositories such as the Enigma Group, which focuses on neuroimaging and genetic data and combining those across studies and making those combined data sets available. The other thing that we've seen is the establishment of many sites, websites that allow for pre-registration. In many cases, these involve, as I mentioned, randomized controlled trials. So sites like clinicaltrials.gov or systematic reviews and Prospero, a good example of a site that's available for pre-registration and systematic reviews. But others are choosing to use pre-print servers and other sites to pre-register their studies. And in many cases, journals are increasingly willing to publish study protocols that allow an investigator or a team of investigators to pre-register their study, including their analytic code. And we increasingly see that in neuropsychology. There are barriers to open science that are not limited to neuropsychology, but certainly do bear on our discipline. In some cases, research ethics boards, individual institutional review boards or whatever they might be called at a local level actually place restrictions on sharing of data. In Canada, where I'm located, it is not necessarily the case that all research ethics boards will endorse or allow for the sharing of data even when it's de-identified. And that's a barrier that we need to try to address at a local level. We certainly also run into, at times, research or reluctance for a variety of reasons. Sometimes it's simply the sense of, well, I worked hard to collect this data. It's my intellectual property. I don't want to give it up. I don't want to share it. That can be particularly true early in the stage of research before all primary study hypotheses, for example, have been addressed in publications. It's also the case that open science is not without cost, cost in both money and time. Preparing data, for example, for deposit and a depository often means creating metadata files and doing other forms of cleaning or removing of identifying information and similar activities. And in many cases, those are not necessarily incorporated into the budgets that are available in people's research grants. Some investigators don't have large research grants. They don't have the staff necessarily that can take the time to make it happen. And there are a number of other barriers. There's a nice article that I referenced here that appeared in the American scientist that talked about how open science isn't always open, that investigators are sensitive to the fact, for example, that if they're working, if they're junior investigators working with more senior collaborators, they may want to promote open science. But if they're senior collaborators are not open to that, there are power differentials that make it difficult. Some of the financial limitations, some of the time limitations may fall particularly hard on junior investigators, on investigators from less advantaged research environments. And we know that the cost of open access publishing is really prohibitive for many scientists, particularly from lower and middle income countries that simply can't afford to pay two, three, four, $10,000 to publish a paper. And these fees are becoming increasingly common, even in the so-called high impact journals, and really place a burden on, particularly on less well-funded investigators. And unfortunately, that may mean that we are open or vulnerable to perpetuating a number of systemic barriers for historically excluded groups. We really need a culture change to address these sorts of concerns that provides funding and support for open science initiatives, being able to include these sorts of costs in one's research grants, being able to have institutional support for these sorts of costs, and providing appropriate incentives for them. Our academic system in neuropsychology as well as other disciplines is really not set up to reward sharing. It's set up to reward individual accomplishment. And until we change that incentive system, it's going to be difficult in some cases to promote this. So the journal neuropsychology that I edit has a number of goals to try to promote open science and are consistent with the American Psychological Association's framework to promote inclusive science. We want to foster an open, transparent, equitable, and reproducible science within the discipline of neuropsychology. And we want to do that by encouraging preregistration, promoting the availability of data, materials, and code, explicitly stating author contributions as a way of equitably recognizing the contributions that each author makes, establishing reporting standards so that all of our, all the papers published in our journal address these concerns, and promoting equity and diversity and inclusion as a general goal, because I think it's fair to say that science really can't be open and transparent, unless it's also diverse and inclusive. So what are the specific initiatives that we're undertaking at the journal to try to promote these goals? Several, a couple years ago, we instituted open science badges modeled on the Center for Open Science and have begun to promote their appearance in the journal. Although we have limited uptake thus far with about 20 papers that have appeared with badges, we've moved to make it easier for authors to request badges and make it a more obvious opportunity for them to show that they have engaged in open science. As part of a larger effort by APA publications, we've also endorsed and begun to implement the Transparency and Open Science guidelines, again promoted by the Center for Open Science. It was very interesting to go through this process with the associate editors and deciding the different criteria that are part of the TOPS guidelines as to which ones we would institute as disclosures as opposed to requirements. And we settled on three that would be requirements in terms of reporting of the availability of code and citations as one example, but others that are simply disclosures as to whether or not, for example, data is available. But I fully expect that over the coming years, we will increasingly move from having simply the requirement of disclosure to the actual requirements of having data available, of having pre-registration. But I think we have to acknowledge that the field is not fully there yet and that it's unlikely that we would be able to institute all the criteria as requirements at this point. More recently, we've also initiated the contributor roles taxonomy author contribution statements or credit. This is a way of allowing authors to explicitly state their contribution to a particular paper. I think this is a really important mechanism for giving credit in an equitable way and being sure that all authors have reason to claim authorship. For too long, I think we've put over much emphasis on being first author or senior author, and then there's everything in between yet as we move to team science and collaborative science, it's increasingly important that we give investigators a chance to state clearly what their contributions have been and we hope that credit will be a way of doing that. This sort of author credit acknowledgment not only promotes openness, but I think greater equity and is something that we've seen adopted by most medical journals already and was one of the reasons we moved in this direction. Most recently, we've started implementing reporting standards in the journal as a way of trying to again promote openness and transparency. In addition to having the transparency and openness section in our methods sections of our papers, we have explicitly called for detailed description of participants and justification of the particular sample to try to move away from exclusively western educated industrialized white types of samples or the so-called weird samples to promote greater inclusivity and generalizability in our samples. We've also begun to require a constraint on generality statement or something along those lines that speaks to whether or not the findings in the paper are going to be generalizable beyond the sample population. If so, why? If not, why not? To be clear that we need to be more circumspect about how generalizable our findings are and promote science in a broader range and more diverse range of populations. The last initiative that I want to talk about is one that we're taking to try to promote diversity in the pipeline for neuropsychology at the level of reviewers and editors because we believe that this is a way to promote more scientists coming from historically excluded groups and therefore broaden the pipeline at all levels and fundamentally believe that promoting diversity in our discipline will also promote more open and transparent science in part because it's likely to be the case that scientists reviewers and editors from historically excluded groups are going to be more sensitive to some of the issues involved in generalizability, the particular sample justification, and some of the methods like participatory-based, community-based types of science that traditionally neuropsychology hasn't made as wide use of. So two programs that we're starting beginning in 2022. One will be a mentored review program for historically excluded groups or individuals from historically excluded groups. This will be a program where we encourage our regular reviewers and our editorial board to identify students from historically excluded groups and involve them in a mentored review process. For students who don't have a reviewer or editorial board member of a local institution, we're going to pair them with someone on our editorial board. We will have them go through a structured process to do a mentored review, a smaller group of mentored reviewers who want to commit to a more intensive program will be identified and provided with some more intensive experience. The goal is to obviously broaden our reviewer pool in the long run and to expose students and trainees from historically excluded groups to the reviewer process, demystify it, encourage them to be able to do reviews for not only neuropsychology but other journals and add them eventually to our editorial board. We will identify all the mentored reviewers each year in acknowledgements and hope to create a student editorial board or trainee editorial board that will appear on the masthead for those mentored reviewers who engage in the more intensive experience. The second program that we're establishing is an editorial fellowship program again for junior scientists from historically excluded groups. This is a program where we take scientists who are in the first 10 years of their career and pair them with one of the associate editors or with me as the editor to go through a year long process where they essentially do a fellowship that involves learning all the steps in the editorial process, shepherding papers through that process, as well as some learning experiences that are geared to both equity, diversity, and inclusion issues but also the scientific publishing process. Each fellow who completes the program will be added to our editorial board and obviously the hope is that in the long run this will broaden the pipeline for our editorial board and our associate editors and as I said a number of times I really hope that the next editor of neuropsychology is selected from one of the historically excluded groups as we try to broaden the diversity of the science which I do believe is a way of increasing openness and transparency. I want to thank you for attending the symposium and the listening to my talk. I want to acknowledge the various agencies, foundations, and others that have been involved in supporting my work in the United States first and then out in Canada after I came up here about seven years ago. I'm sorry I'm not there to be able to answer questions but I hope you enjoy the symposium and enjoy the interaction with the other presenters. Have a great day. Bye-bye. Great and our final panelist tonight will be Dr. Oswald who will talk about open science and his expertise within the discipline of psychology. Great thanks Rose. It's been great hearing everybody's comments from the perspective of their journals and I'm coming at you with a hat on. You can't see it but I'm the chair of the open science methodology committee within APA. I've been a member of that committee since it was initiated in 2018 when Steve Kozlowski was chair and then I took over as chair in 2021. Chair is kind of a glorified title in the sense that we really do work as a team with Katie Corker, Lisa DeBruun, my fellow academic colleagues and then Annie Hall as APA staff. We're all working together to exchange perspectives, ideas, have constructive controversies I suppose and really learn what is happening within APA journals as these practices evolve as the editors are learning themselves. So our ears are open, our eyes are open and I guess there's an opportunity to say please reach out to us if you have any questions. I'll just send my email it's foswaldfoswalde.edu and we'd love to hear from anyone listening who has open science questions who are interacting with APA journals and we just accumulate knowledge and in turn it's a two-way street obviously as we implement our own charge for the journals based on your input. So I'll really talk in broad brushstrokes even though the devil truly is in the details as the editors have reflected on things. But the message, the overall message does drive us forward and seeing at a broad level how efforts are related or not related is really helpful to understanding open science, it's helpful to understanding psychology, it's helpful to understanding the landscape of incentives and resources and processes that are that are going on within the APA ecosystem. Not surprising to you all, I suppose APA seeks to improve research in terms of transparency, reproducibility, and replication. Again the devil really is in the details but as Colin noted it's spurred by attention to the field in terms of the need for replication to have findings that are indeed credible and you know reproducible is a blanket statement just like open sciences. It has many facets I guess that's another point I want to make here is that in the service of transparency there's a whole host of practices that need to be considered in a multifaceted way and that's exactly what the top guidelines are that APA has established within its journals in establishing eight different standards in terms of citation standards, data transparency, I won't list them all but being transparent in terms of research and materials, pre-registering, and so on. So these top guidelines are I guess they're called modular standards which means there are pieces and journals decide on the level at which they would like to adopt those guidelines in line with APA policy so the level could be for authors to disclose whether they used a particular open science practice or there's a requirement at level two that they actually do adopt it and then a level three we'll see as the culture evolves to what extent our journals verifying the practice is meeting some particular standard within the journal but the idea the spirit of it is really important here and that is to develop culture through education through knowledge sharing and through open science itself right because open science materials the sharing of those materials the practice of sharing stands to improve the quality of science it stands to improve the way we communicate to each other and conduct research so it's not just the improvement of the journal article and the scientific outcome of course we want that but it's also improvement of the process of the communication and I mean I'll explicitly acknowledge equity inclusion in that reflection that we want to increase equity inclusion through open science through open communications the fewer barriers there are to research in building this community the better it is not just for again not just for sharing but for improving knowledge and skills we learn beyond the walls of our own institutions and even our own disciplines so the editors have already reflected on a number of points about open science that I think the audience can appreciate but I'll single it out anyway to say that as I'd noted open science is not a monolith it's multifaceted it's a culture as much or more than the practices involved the culture and the practices are are evolving and so is the infrastructure the resource requirements we know in general I think it was pointed out that you know open science sometimes sounds like it's free but it's not free in terms of money resources the time call mentioned the time spent that reviewers spend in terms of their their expertise that they bring to bear if they have that expertise so so we're all educating ourselves while we're reviewing the practices we're engaging in so it's a many faceted enterprise that we're we're nonetheless committed to with all our arrows pointed in in the right direction to stimulate a better research process better research outcome and better researchers again in terms of education so I risk as you can tell I risk repeating myself there but I wanted to kind of highlight what I think is important to the open science methodology committee that that we're here we're listening here we we want to learn and grow the culture as much as the knowledge we want to learn discipline specific concerns and those concerns will evolve as well right that sometimes something that may be shared one way may be shared a different way over time or different strategies for sharing materials and data come up over time as as people engage in open science practices I guess another point I wanted to highlight as as we implement open science is how those behaviors or the the incentives the reputation those can be viewed through different lenses multiple um fruitful lenses so so the journal is one lens uh and and that's clearly important I I serve as associate editor at journal applied psychology working with and for uh Lillian Ebe and her editorial team it's been really great um just fyi but uh it does require as I'm learning in the trenches um the assignment of open science related responsibilities to every part of the journal so you want to implement top guidelines well how what are the what are the what's the responsibility of the authors uh the managing editor the manuscript coordinator the reviewers and and so on and um and so I think you heard every every journal is dealing with with those operational uh decisions which are not trivial um again that's that's where the rubber meets the road and where um you know we have a broad view and a broad mandate um for open science methodology but we want to hear those details and learn across uh across uh every journal as they invest in those details um so I'll say uh two more things um one is uh that uh the open science in addition to implementing the top guidelines the open science methodology committee um has also uh developed in in collaboration with several other professional societies uh pre-registration template um those societies by the way are the British psychological society uh the Leibniz Institute for Psychology in tandem with the German psychological society and and the Center for Open Science uh this pre-registration template is fairly detailed in fact it aligns with jars uh there's sort of a wiring diagram between the components of pre-registration and components of jars and the idea there is to help ensure a more comprehensive research development process uh just like we we take a grocery list to the store so we don't forget the guacamole um we also want to use the template as a way to remind ourselves of the key components as we engage in planning because pre-registration is after all planning and planning is a good thing to focus on and improve right regardless of the top guidelines if open science stimulates the planning the pre-registration template stimulates planning um that that all sounds like good good work to be done in the in the early phase of your of your research um and then finally the road ahead for open science methodology where we're working on appropriate citation for computer software and code this is in line with the credit framework that was mentioned earlier that all parts of the scientific enterprise should be um recognized and um and uh rewarded we should we should be thinking about all this in terms of promotion and tenure committees as well um we're also uh you know api is considering more about the about pre-registration friendly journals ones that are ramping up their efforts even even further or those that haven't started um and want to start how do we uh create that on ramp um as as the open science methodology committee and then we're we're continuing our open science education and and exposure through through various ways of uh broadcasting our efforts through blogs through social media through through short articles uh through communication with with editors um and through authors themselves educating us we can then relay that to others and and further educate in that in that reciprocal relationship I've been I've been emphasizing um so I will stop there I look forward to the Q&A but I hope you all appreciate what all these journal editors are doing the hard work that that they're going through and it really is an honor to support and it's a responsibility to support um and it's enjoyable uh so thank you thank you very much um I'm going to turn things over for the Q&A session to Dr Peggy Christadis who's um in the science director at the American Psychological Association and uh we've been told that people can either raise their hands to ask a question verbally or you can put it uh in the chat feature yes hi everyone um I want to thank our panelists for their presentations and as well as our audience for being here on a Saturday um as Rose mentioned if you have a question you can raise your hand or you can type it in the chat box I also have um a ton of questions that I've kind of been gathering actually over the last year because I I do a lot of work in open science and so I've heard a lot of things so while the audience is maybe um thinking of questions they can type in I can begin with a couple of questions uh that I have um one of the things that I hear a lot about is um how perhaps the transparency of research although it's important is there a dark side to open science and the transparency it requires of researchers um could it open up scientists to bad things like data theft uh public criticism you know like twitter wars and things of that nature or even um misuse of data in the hands of the wrong people who may not have the skill set to work with these data what do you say to people who are reluctant to engage in open science because they have these types of fears and that's for anyone yeah I'm happy to chime in yeah I'm happy to it looks like okay there you are Lillian I really am Peggy and I think particularly sorry I just was saying I think the issues you raise are really important and particularly for people who are not familiar with open science I think this is unfortunately what they sometimes think open science is um I also think unfortunately there there are situations where these type of bad behaviors happen uh threatening requests for data shaming authors who don't share data on social media so I think there's there's really a need for some collegial normative standards for open science and actually Fred and I are working on just a short piece for our professional association around this very issue because sometimes the requests come from people outside psychology who may not have the type of expertise to actually analyze the data or work with the data and come from a culture where it can be very threatening to a junior author to feel like they have to share data so these really real issues I think the base rate is pretty low but all it really takes is one bad experience I think to make people feel really nervous about open science but I'm curious what what Colin and Fred have Colin Fred do you have anything more to add about this dark side I asked Brian Nocic about this as well and he thinks that it's not really that bad that you know there may be an occasional case here and there but usually any kind of criticism is actually quite constructive or people are just asking for maybe additional details or explanation so I hope that alleviates people's fears I can say maybe one one thing about if you like I think I actually wrote about this in an editorial that I that I posted and I think the concern is based on a view I think of science that isn't quite right and that is that one once the research is done or once it's written up or even once it's published that somehow it's done and it's never done it's it's it's a continuing conversation and so I think once we sort of view science in that way then even if somebody you know misanalyzes or misrepresents what we've done that's just the way that the conversation continues so I think we shouldn't shy away from from those kinds of things we should engage them head on and open up the conversation and not be afraid of of where it takes us and I think that's what openness and transparency actually enables so I think the opposite is true the more open and transparent we are the more openly and transparently and and principally we can engage in those conversations with whoever really is willing to to engage with us so I actually think it's freeing actually it's not it's not limiting in that way there's nothing to fear there yeah I think that's a really great way to look at it it looks like we have a question from the audience member um we're gonna see if we can get her to speak Claudia are you able to ask your question you might have to let me see Claudia are you there it looks yeah wait hold on one second guys I can't seem to get her no well unfortunately for whatever reason I can't seem to get that to work so let's go on with some additional questions some of this has to do with more open access related things one person has asked in the past the lack of quality control question mark new open access journals may not have the same perception of legitimacy as established printed journals open access journals do not always have the same level of peer review as established printed journals is this actually true and what are your thoughts on this personally I I lack the experience in working within open access journals I'll say that my experience in submitting the open access journals has been very positive and I say that regardless of the decision that that whether it was accepted or rejected my experience in you know navigating established open access journals has been I've received good good timely reviews I I don't have any I mean these are broad impressions I don't I don't recall the reviews being remarkably better or worse you know that they were they were fairly standard I thought it was you know I think those those processes were were in place there there's also the you know that we live in the in the internet ecosystem now where there's positive aspects and negative aspects for the science right the the negative aspect I'll I'll I'll do the bad first to get it done is uh is that is the misinformation aspect right that that's something did not receive a quality review and and yet still was published and then gets promoted as something that was credible research we see that all the time in peer reviewed research let alone open access that's a reality the positive aspect is that we we also within our ecosystem give papers reviews post publication to Colin's point that science is a continued conversation the journal might be like this you know the fly trapped in amber analogy comes into comes to mind but but with communication comes not only an understanding of that research but an understanding of what research should be following it and I just think we're you know we're all operating off of our histories and our historical ways of thinking about science so so I think with open access journals and new technologies will will you know we're still we're here we are trying to figure this out right but I think the ecosystem is is overall is overall positive but certainly that's certainly that's a concern I'd be curious what others think well if Dr. Leach and Dr. Eby don't have anything more to add we have a lot more questions can you say a few words about qualitative research and what are some of the unique issues or challenges related to open science for qualitative research anyone have anything to say about I'll say a couple of things here sure sure I think and one in particular is that do you have replicas of it being part of the kind of epistemological beliefs of that type of research also qualitative research tends to evolve over time so the idea of pre-registering I think is can be different interview protocols develop and change as you do your interviews or you do your observations in the field so it just poses some unique issues regarding pre-registration and I also think that one of the reasons that open science practices have become common particularly in my sub-discipline is around parking or hypothesizing after the results are known which isn't really that doesn't have the same meaning for qualitative research which often doesn't start with a priori hypotheses so I do worry a little bit about badges and other ways to recognize open science that could be disadvantageous to qualitative researchers particularly in a field like mine that tends to undervalue I think qualitative research so I think those are just some of the initial thoughts on qualitative that's great thank you I have another question in your experience what has been more difficult to do teaching our new generation of psychologists to engage in open science practices or convincing older psychologists like myself who went to graduate school in a pre-open science era to engage in open science where have you seen maybe the most resistance any any experiences with that I can I can say something about that I think I remember those controversies that I mentioned in 2011 when the first one broke of Dietrich Staples misconduct I was actually at a summer school and there was really a generational split and I think I've seen that over and over again and I think there really is a generational divide that younger people recent PhDs and graduate students I think mainly because they're actually engaged in these social media communities that discuss these issues much more often are usually much better informed about about open science practices and have a particular commitment to them and they've kind of had an alternative socialization to research then in graduate school that's probably also because of the way that science has has been popularized in ways for this generation that maybe wasn't true in the past and so I don't know my experience has been a real generational shift and there is there is a bit of a divide and we certainly see that in reviewers and in authors and it's something we're having to work through and you know it is tough and I think I think one thing that's maybe important to acknowledge is of course as you mentioned that you know the training and education that you get at whatever stage is what you build your whole rest that you scaffold your understanding onto that and so it's hard to let go if you let go of that then you let go a lot and so I think we all have to acknowledge that that fundamental changes in how we think about science what we think are good practices are maybe not so easy for a lot of us who scaffolded a lot on top of what we are really learned but I think it's also the case that the truth is that many of us were taught things that now look incredibly naive and wrongheaded and it's also important for us to be really honest and open as we can about common practices that we were taught informal practices that we were taught intuitive ways of understanding and practicing that we were taught that make no sense and I think it's really important on the older generation to be as upfront as possible about what was you know what was problematic in the way that we were socialized and the way that we also maybe even socialized younger people so I think there is a way that that generational divide is going to have to get solved for us all to really get on the same page and work together on these issues. Thank you. We have a question from the audience. They ask what role should tone take in open science conversations? Does delivery matter right? Some people may be very aggressive and demanding that you engage in open science. What do you say to people like that? What sort of tone should we have in order to encourage people to really want to be part of this movement? So I think there are a variety of tones but both positive tones but one tone would be the fact that open science practices accrue to the author as much as or to the researcher as much as to the journal and so don't you want as an author as an early career researcher to engage in these practices that do generally speaking again open science is not one thing but do generally make your work more accessible useful to others. There's research showing that those who share their data those data sets tend to get cited and that then brings credit to the researcher. Also in a researcher handling data requests if you've already shared in the open science spirit but also in practice it's perhaps even routine to honor data requests and kind of tailor what you've already shared to a specific request. So there are other ways to answer that question in terms of tone like one other I'll mention quickly it's just what we've said about social media you know being a being a subtext or a sub tweet for some some of the open science activities that are going on. I would like to think that the more we experience open science as a culture the better we will be at handling open science conversations like data requests and like any conversation that has some element of difficulty like Lily mentioned early career researcher feeling threatened there needs to be multiple people at the table talking not just not just two people duking it out there needs to be multiple minds at play to inform to inform those issues but I think everyone here is an optimist in improving our practices and taking on these challenges. Yes thank you I have another question can open science help or hurt scholars who come from historically excluded or disadvantaged groups I did hear Dr. Yates mention in his talk and he's not here but he did say that you know open science can be for example financially costly so how do we help these groups to engage in open science practices? So I'll just say something quickly here that I just I think there are a variety of angles to this one is simply money you know having money for the to be published in open science journal sometimes there are publication fees so how do we help each other navigate that barrier you know sometimes it's a barrier sometimes it's not it's like you know journals can be flexible libraries can provide funds and sometimes researchers don't know that so you know these are that's the sort of barrier for everyone but for underrepresented minorities first generation college students first generation graduate students they're trying to figure out these the the ropes and they may not come as easily in fact the data probably show they don't come as easily without role models and infrastructure so I think we you know open science is not a one person game we need to support each other and and you know with with our concerns and and need for diversity within our research you know together we need that mutual support to surmount barriers but embedded in that question is is the reality that there are needs to navigate open science you're not you're not born knowing open science so so we need to firm that firm firm that up and and and with the with the concern for diversity equity inclusion great thank you can also add something to that of course so I this is I mean it's like this is my favorite rhetorical trick and maybe it is but I'm going to do it anyway I think this is another example where the truth is actually the opposite of of the way that question of the way we might think about it and that is I think if you're working in a resource poorer environment then actually being very planful being very open and being very transparent is actually an advantage if you know you have few resources to spend then spending them in a more informed plan full open way is actually smarter and I actually think that the the worst abuses in the least open science have actually been performed by the most resource rich places of people because data is cheap participants are cheap and you get into a kind of a factory mentality so I actually think this is another example where you know the opposite is true that in fact fewer resources means you have to be more careful with them and if you're going to deploy them you want to deploy them in the best way possible the most convincing way possible and I think open science does that the other thing about that is and I think this is related to one of the questions in the chat by Gabriela and that is also if we're studying minoritized or hard to reach populations then the same thing applies if there's only a small group of people who were studying we may not want to generalize to everyone you may not be able to replicate 10 times but you know that in fact these participants or this method or this group of people is a precious resource and you want to treat them as a precious resource that they are which means that you have to think very carefully about your methods and what methods work best for maybe that one or two shots that you get to do this study right so again I think these are these I think these concerns are really really important but I think there's nothing inherent in open science that makes those things more difficult and if anything they can actually be resources to study hard to reach or small or otherwise not typical sort of university student populations and they can actually empower people with fewer material resources to do higher quality research. Yeah that's a great point I'm really glad that you're able to see things in a more optimistic way I think people when they see a challenge may sometimes think oh this is I don't know about this is hard and they look at it in a more pessimistic way but like you've mentioned a few times you have to look at it in a more optimistic way and thank you also for answering Gabriella's question in your response. So we have about four minutes left so maybe just a couple more questions. Another question that I've heard quite often is how have contracts with publishers like the one between Elsevier and the University of California that's one example how is that contributed to open science efforts anyone want to tackle that one? Well I it's hard for me to can make this connection I'll start and see if I can get there so I'm I actually shared the the library committee at Rice University where they themselves entered a negotiated agreement with Elsevier in tandem with a I don't know what their formal name is but a partnership of universities in the in the south and you know got saved money for sure on the collections to to then ensure wider access to the extent they would take some of that savings and expand other into other materials that are available. There was a little bit of trade-off with Elsevier and and maybe maybe some other publishers some other publishers they were working with in that some collections were the trade-off was like they they saved money but it was not a permanent acquisition they would have to consistently pay for for access. I don't think I bridged the gap to open to open science there but the goal is to you know in the spirit of open science just generally make science accessible to academics I mean that's what that's what academics is all about and to have a lack of access to publications to the very academics that that may well be contributing to those journals and and yet not have access that that's that would be a sad irony so so it is a it is a it is a struggle but I do think it serves the the spirit and and and the practice open science to have access to these materials you know preprints are helpful as well in and postprints to the extent those are allowable so so that that ecosystem is expanding maybe others have thoughts here. Well we have about a minute left so I really have to ask this one last question um if you if you could look into the future what do you think open science might look like in say 10 20 even 50 years from now um what new open science practices technologies or communities might you see taking hold that's a big question maybe that's something that we should all ponder what will it look like I think we've made tremendous advances just in the last few years so maybe in another 10 you'll see more use of the open science framework more open access all of the all this good stuff so I'm going to hand it back over to Rose and that's going to conclude the Q&A. Thank you for all those engaging questions and the answers that you all provided and thanks to everyone for coming out like we said on a Saturday evening I'm sure you have a lot going on this weekend so it's nice to spend this time with you all talking about open science and how psychology has engaged within that practice thank you to our panelists for being part of this tonight. Thanks Rose. Great thanks for Rose and join it. Thank you.