 Let me be the first to introduce and welcome and greet our speakers for the afternoon. Sarah Emily Wilson and Jesse Fleming are early career researchers from the University of Virginia. Sarah is a special education doctoral student in the Curriculum Instruction and Special Education program at the Curry School of Education and Human Development. She earned her bachelors and master's in elementary education and special education from the University of Florida. Her research interests include pragmatic language development and reading comprehension as well as cooperative learning interventions for students with autism. She's also interested in professional development to support co-teaching and inclusion for in-service teachers. Jesse Fleming is a doctoral student at the University of Virginia studying special education also and before coming to Curry Jesse was a high school instructional coach for a fully inclusive school for students with autism. Jesse's research interests are an improving academic and social outcomes for students with autism through the application of evidence-based practices. Jesse's also interested in examining ways to better understand and apply journal impact factor and open access and pre-print policies and the tenants of open science in the field of special education. And we the program committee and I we specifically wanted to make sure that the early career researcher perspective was provided to this audience was because it's so key to all the work that's going on right now and obviously the types of practices and types of norms and culture we want to have in the next year in the next five years in the next 10 years. And so understanding their perspective their recommendations is super critical for that. They'll be talking about their some of their impressions of what's happened in the past 48 hours what we've seen what what ongoing work is most interesting to them and talking about the future of open science and educational researcher in educational research. So without further ado I'm going to stop sharing my screen and pass it over to them and I might let me double check to see if you have permissions to do that. And I'll pass it on to you. You should be able to do it now. Okay thanks David. Wait one moment well I share my screen that's the the theme of 2020. In 2021 now. In 2021 now yes thank you. All right well we appreciate this opportunity from the center of open science to provide more of a maybe a unique perspective as we close out this amazing conference. We we imagine that there are probably some educational researchers out there who are similar to Sarah Emily and I in that we're still learning about open science and how we can involve and so we hope we hope to speak to a broad audience today but we especially hope that this message will be meaningful for for those of you who are new to open science. Thanks Jesse. Yes we're very excited to be here and are quite honored and you know again just excited to to be able to talk with you guys today and kind of reflect in the past two days. So today we are going to structure our remarks around kind of three broad themes. The first is the why of open science and this includes a bit of reflection on conversations that we've had over the last two days. Cultural changes that we think need to occur for this transformation and adoption of open science practices to be successful. And then we're going to conclude by kind of speaking to early career researchers, middle career researchers and senior researchers about what each of us can do to support this change both in our unique roles and our you know where we are in our careers but also how we can contribute to the work of others and in kind of moving this shift forward. So to you know circle back to our origin stories I would say my origin story was not a love at first sight kind of an origin story with open science. When I began my doctoral program I had no idea what open science was or how it might apply to my work and frankly it wasn't like overly compelling to me. Many of the kinds of open science primers that I was reading discussed replication and the replicability crisis in our fields of education and psychology and as a first semester doctoral students none of that meant a whole lot to me at that time. So there was a lot of discussion of you should care about this and I didn't have enough knowledge to know that I should care about it. But that shifted for me when I stumbled across some readings related to more of the underlying cultural shifts associated with open science and the focus of these readings were much more philosophical in nature and they grappled with messier issues like who has access to resources what do we do to make our work more honest and trustworthy and those were the kind of questions that caught my attention. I continue to catch my attention as a researcher but that's really what drew me in. For me open science was attractive because it wasn't just about how we did our work but also who we are as we do our work. What do we value? Are our actions aligned with the scientific values that we say that we believe in? Are we actively trying to move knowledge in our field forward and better the lives of the individuals we support? Or are we keeping knowledge and prestige? Are we trying to come up with the next newer bigger better thing that may not actually ever benefit anybody? So this is the part of open science the kind of cultural shift behind the work that we do that was particularly compelling to me at that time and remains very compelling to me. So for me as an early career researcher you know new to the field open science isn't just about solving problems of practice in our field but it's about changing the culture of the field that undergirds those problems and ultimately allows them to continue. Thanks Emily. I really enjoy hearing everyone's origin story. I mean it makes me feel like a superhero first of all but it's just interesting to see how everyone finds their way to this movement I guess you could call it but I was originally drawn to research. I started reading about evidence-based practices for kids with autism and that's been a pretty important topic in special education over the last 10 years and as I began to read more about evidence-based practice I found an interesting argument from my future advisor. Dr. Brian Cook about maybe we should rethink how much we know about the practices that we use or maybe we can start implementing you know these open science practices that really aim to make our research more credible and transparent and reproducible and this argument really it struck a chord with me and although I'm very much a novice when it comes to research the movement was very intuitive and logical for me and so I was really drawn to open science for ideological reasons and interestingly enough many early career researchers that I also talk to about this they also find the arguments to be very compelling. As I began to kind of engage in these practices myself and watch more senior researchers engage in these practices really the value has really become cemented in my mind of how important they are. For example Sarah Emily and I were working on a meta-analysis together and we're modeling our paper after a previously published meta-analysis on a similar topic and those researchers who published that paper they posted some open materials in conjunction with their manuscript and it actually really made our job a lot easier as we were developing our own code book and project workflow and so it's this is just one example of many of how we've come across really the value of open science in our work to really increase transparency and support future research. Okay so why open science? When you think about integrity and in credibility they're really the foundation of science and we all know that a foundation is critical to a house we've all watched property brothers right? The good news it's a small crack the bad news it's going to cost 20 grand uh-huh maybe you don't watch property both but it's really the foundation of what we do and without a strong foundation really there's minimal benefit to society and and really the public loses trust in what we do and lately it seems like there's been kind of this movement towards a rejection of science we see it from the public from practitioners sometimes and even other researchers and so one of the best ways we believe to kind of improve that image increase credibility and integrity is to be more open and transparent. Now this is challenging it's hard to be vulnerable and and put everything out there but we believe it's a it's a necessary step to to earn trust. We also think it's important to note that openness relates not only to results but also to processes and so yes we believe that everyone around the world should have access to our our published articles and results but maybe more importantly how we design and conduct our studies our methods analyses and code should also be transparent. And then lastly I I think it's important that we we don't ignore the limitations of science but we actually embrace them I mean being open about how we can improve really allows us to frame this movement as a revolution instead of a crisis a crisis kind of implies we don't know what we're doing uh you know we don't have any direction we don't know how to respond to a situation but a revolution kind of means that implies that we're working together that we have a common goal and we want to improve the credibility of our work and so I think we must address credibility and integrity issues that arise but we should look at them really as an opportunity to change and improve as as a as a research base as a group of scientists hoping to improve outcomes for students and schools and teachers. And so continuing this theme you know I am involved in several you know online spaces really focus on like early career researchers researchers from historically marginalized backgrounds in academia and kind of discussing you know what are problems that exist within academia what are the barriers that people from communities that have been not as included in academia in the past what are the barriers that they see as inhibiting them from like fully accessing academia fully accessing kind of this this space this research base that we all exist in and a lot of the barriers go to equity of work democratization and that idea that things have been there's been gatekeeping things have been closed and I think open science can move beyond just addressing replication but also again address this this kind of culture that we have and this to me is an early career researcher again it's really valuable it's really compelling to me and it's really important so how do we how do we shift the work that we do and the culture of the work that we do to ensure that our work is better for the people who come after us and so I think again open science practices can speak to this and underlying issues that we see within our field can be addressed by open science by supporting equity democratization efficiency and the reproducibility of our work by adopting open science practices and so when we think about specific open science practices I heard this touched on in a couple different sessions different practices can contribute uniquely but also build on each other to increase the equity and democratization of our practice open access preprints open data open materials these all support more equitable access to research and research materials for those without access so this includes other researchers practitioners policymakers and frankly anyone who doesn't have a university library system that has removed that paywall for you and I think that's that's really important and that takes an important step towards equity within our work and another step towards equity and democratization I think is through registered reports I think there's a really unique opportunity through registered reports to diversify who gets to ask questions and what questions that they get to ask a big problem at least in the field of special education is that the majority of large grants only go to a handful of researchers at a handful of institutions and not only does this limit the representation of our sample but it also limits the representation of the background perspectives and philosophies of those getting to ask research questions and so registered reports helps spread and share expertise before the research process truly begins and it contributes not only to the democratization of research capital but also makes our work more efficient and there is a ton of pressure and I feel it is an early career researcher I'm sure you all feel it regardless of where you are in your career to be publishing on your independent line of inquiry publishing new and novel findings you know what are you contributing to the field that's new however I ask myself a lot it's like is that actually what's best for our field is that actually what's best for the students that I say that I support is that the most helpful for them so so thinking about you know what what actually makes our work more efficient and I think again a lot of these open science practices support that and our work is more efficient when we share data and code others are able to ask research questions from our existing data sets and apply our code to their new work it becomes easier to replicate studies when we engage in practices like open materials and pre-registration when I was originally conceiving of my dissertation this is prior to COVID and it's since shifted I wanted to do a conceptual replication of an intervention treatment package the materials though needed for the training data collection intervention implementation everything was available but it was behind a long process of emailing networking and applying to get those things from the original authors and so COVID aside I'm not sure I would have been able to actually pull it off because there were so many barriers between me as a researcher and those materials and that it just makes me think about like the knowledge we can move forward in our field with if these things were more available to everybody if there was more equity if there was more efficiency in our work it's a way of passing on expertise and knowledge and resources to newer generations of researchers that I think ultimately makes our field stronger but also makes the lives of the practitioners family and children that we support I think it makes their lives better as well and that leads into that kind of final component of the efficiency of our work and making the research to practice gap smaller and making our work getting into the hands of practitioners and the people in the field making that a more efficient and quicker process okay so we're kind of moving into the second step of our outline here the cultural shifts that that need to that we think need to happen and this first one really touches on some barriers we've talked about a lot of these over the last two days but we think it's important to touch on them again but open science clearly be you know over the last two days we've seen that it's becoming more common in education research but we believe that you know maybe through additional supports and incentives we might be able to improve uptake of open science practices so we have their education specific training and resources maybe resources for diverse educational methodologies ethical considerations when thinking about data what's a student data maybe how open science can support you know the research to practice gap and and even how we can maybe engage in direct or conceptual replications across different classrooms and students so you know these are just a few examples of maybe where we could have some additional trainings and supports there definitely has been some resources developed recently but we have found that dissemination and access to those recent resources can be a problem um Sarah Emily and I have really talked about that second one a lot integrating multiple practices we've really struggled to know how to sustain and integrate multiple practices across the lifetime of a paper and you we understand for the most part how to to implement and use you know these one-off different open science practices but it might be helpful to have more resources and guides that really explicate to researchers what should be done in each phase of a research project we also believe that mentoring is key to this cultural shift senior researchers can do so much for early career researchers so that we can better understand and navigate this community mentoring can look very different depending on the situation but it could include opportunities to do one or more of these practices together collaboratively provide opportunities for service within the community provide resources or even kind of initiate maybe more of a grassroots program like a journal club and lastly we we view the need of we need buy-in from all stakeholders journals and publishers institutions professional organizations it's really going to be challenging to have widespread change cultural change without buying from all of the important stakeholders in our field now moving down to that second point incentives we also believe that incentives need to change or you know shift to match the changes in research output that we'll be providing promotion and tenure committees can consider open science practices hiring committees could add open science practices and their searches and really institutions and funders could do more to value diverse output such as open code open data sets and even multi-site collaborations I recently watched a presentation from Anne Schiel she's a an early career researcher in the Netherlands and her presentation I really like this metaphor was vanguard or cannon fodder early career researchers on the front line of open science and I I think that metaphor is really compelling are we are we pushing researchers especially early career researchers to the front lines and asking them to change the research culture without changing really how we support and evaluate and incentivize that research you know and so when you think about it our researchers going to be leading the charge for change or will they find themselves you know without employment without tenure as the the culture failed to value value their work so we really believe especially for early career researchers that incentive structures need to change to value open science and diverse output as we move forward along with incentives to to accomplish these changes I think we have to also consider effective aspects of our research or culture as Jesse mentioned there's kind of this inherent vulnerability to openness and transparency particularly given the current culture in our field a written newsletter on open peer review with Brian Cook a few weeks back and that interest that literature was just so interesting to dive into because peer review in many many ways can embody a lot of these cultural issues within our field I'm sure experience getting peer reviews back that halfway through we kind of take a pause and are like well that no longer feels constructive towards bettering my study um and you know I'm sure we could and there's feelings involved in that right of like this no longer feels like this person's trying to help me this feels you know personal or somehow like you know that they they have a message to send about the kind of work that I do rather than how I just work that work and I think that that that kind of you know emotional effective component can really inhibit folks's willingness to put themselves out there and try on new things including openness and transparency and I think that you know for early career researchers that that's that's a huge barrier for early career researchers to feel like yes I want to go try this new thing when already I might be getting feedback that again is critical feedback is great it's constructive and so I think it's important again to just really consider how effective cultural aspects may need to shift in order for us to truly adopt open science practices across our field and to to allow this kind of effective shift in culture to occur I think we need two kind of big things we need to allow time for change and we need we need to make it safe for participants to engage in these practices or for researchers to engage in these practices so and allowing time for change people are going to make them safe and it takes time to adopt new practices it takes time to develop expertise in the practice and if we are committed to openness and transparency there needs to be patience and support and this adoption process we need to actively support encourage and mentor growth and I think again that's particularly important for early career researchers to know that if I try out a pre-registration and it doesn't quite work the first time that you know I'm not going to be kicked out of the open science club or something that there's a community behind that's really focused on supporting and growing expertise in these practices and I again I also think that we need to make it safe for people to participate in it we need to allow researchers to participate at their own pace and kind of reject that all-or-nothing mindset and we need to allow researchers to participate in different ways we need to have more space at the table again I really struggled with this and a lot of open science materials I read very early on we're so much of it was written from a positivist lens or kind of a post-positivist lens that that's not the kind of work I do and that's not necessarily the parent dime that I align with and so I was looking and I was eager to find more work and more dialogue about well how does someone like me who approaches my work not in that way how do I adopt these practices where do I fit in in this narrative so I think there needs to be space for folks who want to do this work maybe for different reasons or in new ways but again so that the shift can occur across our field not just in a very narrow type of educational research or a very narrow educational paradigm and being kind and feedback again I think that goes back to like being supportive and encouraging of like when people try something and it doesn't quite work or maybe they need to do something differently the next time looking to be supportive and grow expertise within the community rather than baking and then finally I think there needs to be caution when comparing scholars with transparent work versus less transparent work making sure that we are embracing kind of the risks that come with openness acknowledging what it takes for someone to be open respecting that and saying that you know there may be a difference in not necessarily work quality but outcomes of people who are like I'm putting everything on the table you can completely see what I did versus outcomes of you're not really sure how people got to where they got so again I think the cultural affective shift that needs to happen is going to be huge for whether or not we actually see adoption of open science practices becoming more widespread or kind of staying within a smaller community yeah we actually added this be kind and feedback that was a ulterior motive for us so just a reminder so now now we're going to move on to the kind of the last portion here so don't fall asleep quite yet kind of what is your role in the movement and when thinking about early career researchers when thinking about my own kind of foray into open science I actually think about a few years ago a friend of mine invited me to go to this gym with him and I'm not really the gym type I don't exercise that much which is bad don't don't let that be the message from today but I finally agreed to go with him and we go to this gym and everyone at this gym is very athletic they have lots of muscles and much larger than me and they are very passionate about what they were doing at this gym and then on top of that like I look around this gym and there's just all these tools and yeah equipment I didn't even know where I didn't know it was exercise equipment I had no idea what it was and that you could use it for exercising and you might be able to guess what type of gym this is but I I think back to that experience and early career researchers really sometimes feel like this we come into this you know open science and there's a lot of really intelligent bright people who are very passionate about this work and then you look around and there's these these tools that we should start using but you're not exactly sure how that that fits into your own work and so it can be very daunting and Sarah Emily and I really get that we understand that feeling and that's okay it's like if you want to be successful with this you have to feel you have to be okay with feeling vulnerable and so I think it's also important to recognize that open science practices can take a lot of extra time and resources and even though they sometimes appear to be very straightforward at first you know things like pre-registration and open data those practices are are pretty complex they involve multiple steps and and oftentimes require additional training and so there are these disclaimers but even with that we we do believe that open science practices can benefit early career researchers and there's a place for us in in this there's a role for us and so you know open science practices can really distinguish your work especially when applying for jobs it can lead to more credible and reproducible work and have many benefits like broad dissemination and increased impact so there's definitely some benefits to getting involved so what can you do today I I say this too much but I I think trying something new is really important there there are many practices such as posting pre-prints or sharing your open materials that can be done without much training or time and you could also get creative with some of these practices you know we talked today about early career researchers using their dissertation as a registered report you know which ensures publication and I'm I'm sure there's a lot of researchers on on this this call today that could talk for a very long time about the struggles of getting their dissertation published right and so there are definitely advantages to kind of using some of these these tools uh for our good so um we also recommend that early career researchers connect with others who are interested in this topic engage in conversations you know talk with your peers talk with your colleagues get on social media attend conferences which you're already doing but don't be afraid to ask questions you know as as the future generation of researchers we really believe that our voice and and perspective are important to moving this moving open science forward within education um sorry I'm seeing some really like great comments in the chat and I want to like respond to them but I also know that I'll totally get distracted um but one of the things that I saw pop up was this idea that like we may not be able to adopt all practices at the same time that may not even be ideal for a project that may not fit the project um and I think that you know lends itself really really nicely to this next kind of point of like how can mid-career researchers who maybe have a little more expertise a little more practice than this um how can they uh support early career researchers in um adopting open science practices and how they can support um kind of this cultural shift in this movement towards these practices more broadly in our field um and I think modeling that you know modeling this idea of I'm going to try practices on I'm going to use these practices for this project and then I'm going to add one or subtract one for another project because it really doesn't fit quite right um so mentoring and modeling those practices I think is really really key um you know from from the perspective of an early career researcher it's that kind of middle career researcher feels attainable to me it feels like oh I can see myself in a few years kind of being closer to to that point in my career collaborate with us and collaborate with each other um try something new again you know I think Jesse mentioned that we say it a lot but I think it's really important uh and you know circling back to the effective component I think encouraging each other encouraging us um encouraging the field to embrace these to explore these to um not get uh stuck in what we know to not kind of get that complacent of well this is worked you know I've worked my doc program will clearly work for my advisor so I'm going to just keep doing the same thing uh but encourage yourself and each other to to resist that complacency and to um try to push yourself in the field towards um towards having that kind of cultural shift in adoption of practices um next there has been a lot of dialogue I think over the last two days about training and I really think that there's an opportunity to integrate practices into courses and assignments for doctoral students um if this is in you know doctoral training seminars or an other kind of doctoral level classes that you're running um or instructing and then I also think there's an opportunity for a middle career researchers to um advocate or actually enact open science practices within doctoral milestones so that might look like as a part of my doctoral training program I have to do a preregistration or I have to make my open date my data and my materials for my dissertation study that it's required to be made open that that's a component of my dissertation as a component that I'm assessed on by my committee so I think there's ways to embed some of these practices to help normalize them um a little bit more I think middle career researchers can also start those start journal clubs the reproducibility activities and I know open the open science center has done before and I know Jesse and Brian are both involved in at EBI and then engaging in like pre-review okay thanks ceremony so this is the last slide you made it we saved the best for last um when I was eight years old my my neighbors his dad got fired from his job they uh this isn't a sad story he got a new job and everything was fine but um he I remember this phrase being told to me and I it didn't make any sense to me but they're like yeah he lost his job they they got new owners and you can't teach an old dog new tricks and and it didn't make any sense to me I was like why can't an old dog learn new tricks that doesn't make any sense but as I've gotten older you know even now like there are some tricks that I just can't do you know uh those uh you know hoverboards I just I can't I can't ride them I I hurt myself so um I I understand why there might be some senior researchers who struggle with open science you know they're being asked to change the way they've been doing things for many years and uh they're being asked to open up their data sets you know asked to be more transparent uh about research that they've built their careers around and that that's not an easy change you know and so uh I as important as I think grassroots efforts are uh you know efforts from early career researchers and others I I really believe in those I I think senior researchers really have to step up uh to the plate where they you know where they have influence at universities at funding bodies you know journals there they're editors at journals they review grants and and advocate for for the change um and really uh working together all three stages you know anyone can really do any of the three but we thought that these were kind of the steps that were most appropriate for each stage but uh you know working together uh everyone involved we we really believe that there's some really positive change that we can make if we kind of if we all work on you know where where we have influence and and improving what we can so that is the end thank you for listening and staying thank you Sarah thank you thank you Jesse um everyone please join me in sort of a round of applause the zoom doesn't quite uh imitate that perfectly but I do love these applause emojis I really appreciate hearing your perspective um I I don't know what others uh we'll open it up for questions in just a moment but I'll take the uh kind of host prerogative to um to start it off I was really struck by the uh you know the cannon fodder or van or a vanguard or of course you know going into the gym and having some of that what what do I do first here and I think that's a perspective that's a lot of people can relate to it's not early career researchers it's anybody stepping into a community a suite of activities for the first time it can be it can be scary it can be daunting there's there's a lot to take in and um a lot of the themes that you touched on of criticizing with kindness you know a lot of what we're asking folks to do or trying to model ourselves is a new step and it's the first time everybody does something for the first time and so recognizing that that type of criticism has to be um given with that in mind is the only is only a successful way to to to really move forward if we don't and we're asking folks to do these optional things and that just leads to punishment uh Brian shared that a couple of stories and in the chat also we're going to we're going to peter out we're not going to be successful um I'm wondering of of the initiatives or projects or activities you've seen in the past 48 hours um today and yesterday's sessions what do you think is most what do you personally have the most motivation to um continue working on tours or being a part of or what what do you think is most um what are you most optimistic about from what you've seen recently um either Jesse or Sarah uh I can I can take a uh you know crack at that question um I think one of the things that I've been most impressed with over the last 48 hours is not necessarily one single project or like one specific talk um honestly that I was just most impressed with the consensus regarding the problems um you know I think that there's a lot of times in you know when you have a group of people and people can come to the table like this is the problem no this is the problem no this is the problem and that can really inhibit any form of momentum moving forward because you can't even agree on what the things are that you need to be addressing and I think that I was really impressed with how consistent the dialogue and discourse was across sessions of that okay we have a good agreement on what we need to address and that feels like that we could actually then move forward with kind of solutions oriented conversations because we at least agree on the first part of the conversation and sometimes that's the hardest um so again not necessarily a single project or discussion that I've heard but more of just the the continuity between discussions that we really do feel like most of us are on the same page uh which I think is a good place to be yeah I'm I'm gonna echo that I I think the last 48 hours for me have have been very impressive a group of you know very diverse scholars from around the world coming together you know I I probably shouldn't say this but um I wasn't a full believer in the un-conference at first but I think it's been very effective to to allow individuals to think together and to collaborate and work together um and so it's been very impressive that you know we as this wide range of individuals can come together and really begin to to solve some of the problems that we see uh secondly the community has been very open and welcoming I didn't expect otherwise but but everyone has been really really great to work with and so yeah I've just been really kind of impressed with the opportunity to collaborate and work on problems together and that's you know thinking about education there's so many different you know we're in special education but there's so many different schools and and thoughts about education and that we can all come together and work together and agree on something that gives me you know hope for what we can do for the future yeah I think absolutely I echo that just there's a lot of um optimism to be excited about um I'll open up to others for questions or comments then we can also transition to sort of a round table um format a little bit of a sort of a reflection on um on the past few days and then what the next steps will be yeah I so I don't want to take a jump to the front of the line but um I tried to use enough wait time before I chimed in um I was just thinking back to a conversation I had with someone a couple of years ago and this is I'd be interested ceremony and Jesse in your perspective but anybody's thoughts on this around I think one common element uh around the different open science practices approaches that that we've been talking about perhaps some more so than others but they all take extra time extra effort um it for some of these practices it's ongoing time and resource commitment um I I just worry about balancing that or or how do others perceive balancing that with the almost insane level of of commitment that academia asks uh and not just academia uh profession being a professional uh around being a scientist I I so wish we could slow down things a little bit I I could be just way off base here but in my head I think of of open science is almost an effort to let's slow down a little bit and instead of just rushing the heck through everything and moving from one study in one article uh to the next as fast as we possibly can because the goal is to have the longest CV um but let's slow down and do it the best and most transparent and most credible most replicable uh way that we can and it just takes longer um I don't know I I feel some of that tension I think I feel less of it because I'm the good fortune to to be tenured and um promoted and and and so there comes a certain freedom with that but I'm still been in part of this just I don't want to call it a rat race exactly but the system where it is just boom boom boom next thing next thing next thing and and I wonder how we carve out a space for open science in that I don't know if I necessarily have the answer to that but um if you do love it I I I echo that sentiment you know and I think that even being three years into my doctoral program I already have experienced working with uh certain collaborators that I'm like maybe I will not collaborate with you again because we have a different idea of the pacing of our work or the quality of the work that I feel like I want to hold myself to versus this like you know we're just going to kind of slap something together to have you know to keep moving quickly um so I think that's for me like being like feeling like I can have that conversation openly with more senior professors I think is the first step into like oh that's that's something that's okay for me to feel it's okay for me to think it's okay for me to think about um I can choose to um you know maybe watch how others are operating their work and I can choose to operate differently I can choose to slow it down and I can choose to be more intentional about this um and I think just as silly as it sounds just knowing that that's an okay thing to explore and allow myself um not that I you know it's not like giving permission but again just like knowing that that's an okay thing to think and experience um honestly I think is a good first step in trying to kind of shift that kind of rat race mentality of how quickly can you turn things out every day I still talk when I'm muted um I don't want to call Colleen out but I thought the the comment that she made in the in the chat was was just excellent Colleen are you on do you mind just sharing your thoughts on that yeah no it's I think about this every time I go to design a study right like I have moments where I'm like okay we could design the super quick study just like get it done and even when I say that it's not true anyway so I find this time this timepiece I keep coming back to as well um but I think as a field right if we think about having five studies of you know borderline quality in terms of what that what we learn from that compared to one you know really well done study like it's the idea of meta-analysis right like if everything you put in the meta-analysis wasn't well done it doesn't matter that there's this many studies now right like um and so I think the time investment piece and the pacing it's something I really struggled with in in my lab especially advising my students because you know they've got five years to try to have some papers on their CV and uh five years actually not that much time uh given the publication process so I think there's a real a real conflict between time spending the time to do this type of work and the incentives to to publish um so even though I think lots of open science things will actually help like open data right if I can tell my students go to this data set somebody already has like in the long run that should actually save time so and I think that same message around it's it's better science it contributes more to the one really well done hopefully open study contributes more to the evidence base I I think there's a similar case can be made for one's career too that at least in my experience um those higher profile but hopefully that corresponds with with higher quality more meaningful studies that's what I think really helps uh progress you up the the career ladder more than just pumping out I there's some of it probably has to do with with quantity and just counting things up but but I think the the quality aspect sometimes gets undersold and and I I think there is a there is room for the the value of of really good and and open science to be done I think this is tangentially related to that um but I think as a and another kind of you know ask as an early career researcher what I would ask people who are more senior in the field is to especially as you are thinking about how to support your doc students you know there's a lot of pressure as a doc student of like and my advisor you know Bill Therian is in this room so I'm saying that you know that there's a lot of pressure to balance between what Bill you know talks so much about and I work very closely with Brian and these things that I want to do and value in my practice and then I walk into a doctoral seminar taught by another professor who does not value those at all and has very different expectations for what success looks like in research and it can feel really conflicting of like I'm getting two very different messages and so I think there is also a role for middle and senior career researchers to also have that dialogue within their departments about what is the messaging given to doc students especially within doc students seminars and you know can we have more of a common or shared kind of charge to doc students or a shared message to doc students just personally it's that has been a challenging aspect to kind of navigate as I have one you know the person leading my doc student seminar is very focused on you need to have blank publications by the time you leave or you will not get a job versus you should be doing high quality intentional work and it's okay if it's only you know a few publications but they're really really strong that that's what matters and so you know having those conversations above a doc student level I think would be helpful so in our last few minutes I wanted to pose a question to to everybody and I won't call on everybody but feel free to think about the response or chime in but thinking about what what next what's the one thing that you are going to sort of take away or try to do next after this and I want to first just to get the ball rolling just for a second but there are a couple of hackathons that I really want to continue on creating some guides and we're not going to call it this but they what doesn't work clearing house the conversation that happened there I think has a lot of fruitful momentum and trying to figure out the best way to continue and grow this community conversation that those are some of the things that I know I'll be thinking about over the next few days and I'll pose that to everybody in our last few minutes too. Can I add one thing um this is sort of what I was thinking of saying earlier but I think one important step is to realize that for many of us we are the open science person in our communities like we have all come together from all over the world and we're like learning from each other and this is great this is the open science community but then we go back to our departments and we are the person with the most knowledge even if we feel like it's nothing and so I think just in general a good step is to like take whatever you've learned here and talk about it with your colleagues bring up these concerns um that sort of thing so it is nice to have this like central focus and I'll let us get back to that but I also want us to recognize that even though we might not feel like the open science person like in my experience it's very easy to all of a sudden find yourself in a room like I know more about this than everyone around me and that's a little bit shocking because I don't feel like I know anything um so uh definitely I think that's an important part of continuing this mission is to just like raise awareness of what this is and why it's valuable um but in terms of like next steps um I don't know how other hackathons have went but we've um and at least the group uh with Ethan Brown and such we we've exchanged emails and we're on Twitter to connect and sort of keep this idea momentum going forward I don't know if other people have other ideas I I'll echo that idea of trying to to keep this going and keep the momentum going I um I was very interested to see what what especially with it being online in the middle of a pandemic which is obviously unfortunate on lots of different dimensions but I'm very interested to see if this continues in the direction of of SIPs or the Society for the Improvement of Psychological Psychological Science and to to try to help this grow and continue and and look forward to both some perhaps some ongoing activities but then to a bigger hopefully in person meeting next year and continuing to grow this because I think that's um we we can start it but but it has to to grow to really have some some big effects if it isn't just to remain kind of a niche thing that that a few of us are into hopefully turning uh getting rid of even the phrase open science and just it's successful it just becomes science the MO of science everybody thank you so much for your participation for your energy um as Brian just mentioned it's you know it's tough out there so thank you for your uh thank you for your support thank you for your contributions um this will continue so and now and that's because of of of you so thank you so much