 Hello and welcome to Esmar Conf 2023 and the barriers to open synthesis and how to remove them session. This session was pre-recorded because of speaker availability and you're now watching on YouTube. Automatic subtitles should be available now and will work hard to get these manually verified as soon as possible. If you have any questions for our presenters, you can ask them via the at ES Hackathon Twitter account by commenting on the tweet about this session. If you registered for the conference, you can also comment and chat with other participants on our dedicated Slack channel. We will endeavor to answer all questions soon after the event. We would like to take time to draw your attention to our code of conduct, which is available on the Esmar Conf website at www.esmarconf.org. So thank you all for joining us today for the session. I am Emily Hennessey and I'm an associate director of biostatistics at the Recovery Research Institute at Massachusetts General Hospital and assistant professor at Harvard Medical School. I will moderate today's session. We have a wonderful panel, a set of panelists for this session and I'm going to ask them each to introduce themselves. I'm starting with Tim Errington. Great. Thanks, Emily. Thanks for the invitation. So I'm Tim Errington. I'm the senior director of research at the Center for Open Science, which is a nonprofit that's based in Charlottesville, Virginia. I'm going to form my life. I was a preclinical cancer biologist, but since I've been at the center for about 10 years, I've become a meta scientist kind of studying the scientific process. I think one of the things I'll mention that probably hopefully comes through as we continue this panel from my perspective is what we do at the Center for Open Science, which is trying to increase openness across that whole life cycle of the research. So not just data, not just publications, but all the processes that play a role. And that's a really huge part of what we're trying to do to serve our mission. Great. Thank you. Okay, I think I'm next. Thank you, Emily, for the invitation and it's wonderful to be on this panel with colleagues and I'm very much looking forward to it. So I'm David Mower. I'm a professor of epidemiology and public health at the University of Ottawa, which for those who don't know is the capital of Canada. I've conducted hundreds of systematic reviews, developed the first iteration of Prisma, which is really getting at the notion of transparency. And I've been a co editor in chief of systematic reviews, the journal systematic reviews for a decade. And I've led and lead and the member of lots of different projects on trying to develop a culture of open science. Right. Thank you. Oh, I think you're on mute. Okay. Oh, sorry. Thank you for the invitation Emily. It's nice to be here. I'm a professor in the School of Public Health at the University of North Texas Health Science Center. Since 2010, I have directed an NIH supported synthesis project called Project Integrate. In 2009, obtained individual participant data from original investigators and synthesize data to provide evidence regarding RIF I call interventions for young adults. My team and I have also developed statistical methods to combine individual participant data from heterogeneous studies. Last year, we competed for the NIH data works challenge, and we were honored to be a finalist team. Thank you. Hi, I'm sure. Thank you so much for the invitation as well. I'm an assistant professor and the director of the baby lab at the International Research Center for Neuro Intelligence at the University of Tokyo. And my bread and butter is really being a developmental psychologist, but I see open synthesis as an integral part of my work. I, like many people in my generation of psychologists got into the topic because I grew up in the replication crisis. And I realized that it's important to synthesize over evidence, especially in the field with very small and noisy samples like infant studies are. And so I started a platform called MetaLab together with colleagues, which is an open synthesis platform about meta analysis on cognitive development. Great. Great. Thank you all for those wonderful introductions. This is a panel discussion so I am going to try not to dominate the conversation I did just want to set the stage so that everyone who is listening can have a good kind of base understanding of what we mean when we say open synthesis so I'll briefly go over what we mean when we, when we use the term open synthesis. And hopefully if I am incorrect in my assumptions here, the panelists will gently guide me to the correct definition of open synthesis, but then also really give a brief overview of who maybe the key stakeholders are. And when we're talking about this. And so Neil Hadaway and colleagues recently put together a wonderful graphic on kind of the different components of an open synthesis framework and so essentially if we think about it if you're a primary researcher and you think about what it means to be, you know, an open scientist of a primary study, it's really kind of taking some of those key elements and bringing that to the synthesis world, the systematic reviews and meta analysis. And so there's lots of different ways that synthesis can be open and that's through kind of open collaboration with with folks kind of across different nations different disciplines on having there be opportunities for open discovery through research, search engines that enable bibliographic searches of databases that are easy to use and navigate and provide access to you know all the potential primary studies that could be a part of that process. Obviously when we're talking about open synthesis, the emphasis, some of the emphasis is going to be on having open methods and transparent reporting of those methods and the data that comes along with that. So there's kind of lots of pieces to this process I'm not going to go through all them because it could take the entire, the entire conversation basically and I really want to hear from our esteemed panels but essentially what I what I want, folks that are starting this panel with us to the idea is that the idea behind open synthesis is that we have a synthesis ecosystem where where there's a free exchange of ideas and data and methods and it's it's very transparent it's replicable. And ultimately it kind of will hopefully improve the process for everybody involved. So if you have a question or if you have a mind, kind of jump starting into the conversation what I wanted to know from from this set of panelists with diverse experiences but all with expertise in this area. You know why is it important for the field, you know broadly of science as a whole and perhaps for your own discipline why is it important to have open synthesis. Maybe I'll just jump in with one point Emily. Maybe it's a different perspective from the other panelists I don't know. But I think one of the reasons is to respect and honor the views of patients. About having their data shared and the research that's being done shared and my discipline is medicine. And there is a growing data to suggest. If not indicate that patients are have strong views about the data that's collected by them as part of a research project and particularly around randomized trials. They want that data shared for additional studies and that the data can be used by other researchers that really important I think. Let me chime in here. I think the reason the way I think is that the reason why patients wants to share the data for new knowledge is, I guess, in it can be summed up as like we want to speed up scientific discoveries right. If data is shared at a faster rate and more widely, then we can actually speed up the knowledge discovery and also we increase transparency in data and strengthening public trust in data. And also, in the end, ultimately, it saves time and resource for the field. So open synthesis, the way also Emily pointed out, benefits all stages of data generation like the ecosystems and ultimately the public we serve the public as a scientist. And in my case, for Project Integrate, we started to share data and computing codes in a public repository called Mandalay data in 2019. And we now try to share all the data and paper when a paper is published Mandalay data tracks how data are being utilized. For example, the data and our code we shared in 2019 have been already used almost 900 times. And we also have a few citations stemming from data reuse from our data that we shared. And our data sharing practice is consistent with the fair principles, findable, accessible, you know, operable and reusable. So there can be a lot done to promote open science and open science benefits everybody. That's what I wanted to say. I'll add on to that. So first I think what David mentioned first about respect the patients resonates with me a lot, because we work with babies where the parents come into the lab they're normally normally developing babies but still parents put a lot of effort into coming they're very proud of it so they're often very happy now that we asked them whether we can share their data to to agree to that, as long as it's anonymous of course. And then a more researcher focused view is that often, especially infant psychology labs are often not so well resourced so even though we want to collect bigger samples it's very hard for us. So this is why open synthesis can be a way to collaborate or to build on past evidence without what basically doing something that is not possible for us. And the third point I want to bring up is that, in addition, what you said is that I think open synthesis of really important against biases we have in the literature for instance the weird bias like Western educated industrialized rich democratic country bias that we have and open synthesis is at least one way for researchers working in countries that are not weird to get access to data without having having to pay for them. I'll just chime in. I think that's the all the points raised are great. I think I'll summarize my point of view and the simple terms which is to help us know what we know and what we don't know. So I think, you know, we're always hunting to try to keep synthesizing data to get the best estimate at any point in time so it's a nonstop game right we're always chasing ourselves, always looking for that next study. The open part is critical and a lot of what you were just saying show right like it was really important that we also know what we don't know, and we shouldn't just like look at small studies, or look at specific samples and quickly jump to kind of broader conclusions we really do need to kind of think about everything that quote unquote worked and didn't work right not just you know, a narrow one so this gets a little bit into the challenges we have in terms of doing that. But that open piece of really making sure we're looking at everything really helps us kind of, I think collectively figure out where we should invest our resources to kind of keep, you know, honing in on that picture. Yeah, thank you. Emily, I would just add, and another reason that I noted was the notion of reproducibility. So in medicine, you know, we have a sort of a synthesis factory. And typically those synthesis are used in what we call clinical practice guidelines, which, you know, for example, it's, you know, if I go to my family physician, they may be recommending a particular treatment or a particular screening period, based on a system of review that's been included or used as the clinical practice guideline. But so the question really is, if those reviews are not reproducible. Are we going to run into significant health care problems. And if we take the open science collaboration seminal paper of 2015 where they're looking at primary studies not synthesis. I don't think those results are anything reproducible with systematic reviews or knowledge synthesis. That is a huge health care problem. Yeah, it's a major, major problem and really, you know, all the answers have kind of demonstrated to me, you know, the different levels of stakeholders and why there's something in it for everybody for there to be open synthesis. And also I think David back to your first point on, you know, something that that I think is a misconception in this area is that individuals participating in these studies on, you know, are concerned about their data being shared but actually you know what you've shared is that they, they want their data to be shared because they kind of see the larger, they can see the larger mission of what this is about. Yes, I think there's, there's two, at least two studies that I think are very credible. Michelle mellow did published a very nice piece as did Kim. So there's, I'm aware of those two in medicine. No doubt there's others that I'm just not aware of but there is a growing perception of, first of all, patients want this. And second of all, I think they're often the stakeholder that's forgotten, but central. Yeah. Yeah, thank you. So, so knowing you know the importance of open synthesis what what from your perspective each of you kind of individually what do you think needs to be in place for open synthesis to occur and this this can be at, you know, several different stakeholder levels it can be more technical or more kind of practical I would just love to hear your thoughts on on on these things that would that would help this happen. I apologize for jumping in. Oh, that's great. I think, you know what needs to be in place I think lots of different things need to be in place. I think the first one is is a culture is a culture of open science across research teams I think that's particularly important. I think we also need to have in place for example, data management and sharing plans. In medicine this is quite complicated because we do clinical trials, but we also do in vivo and in vitro research and so they require quite different data management and sharing plans. And so, for example, in Canada they've recently mandated, they're sort of rolling out the mandate of, of requiring data sharing. But there's not a lot of plans there that templates that people can follow. And so that that's going to become a barrier. So we need to close that barrier. I think for individual patient data men analysis we obviously need to have consent in place. That's particularly important. And then I think journals and funders need to require it. So again, in medicine, I can think of only really two journals, or publishers, the PMJ publishing group and the Public Library of Science publishing group. I think they're the two strong publishers that have strong requirements for data sharing. It's possible that within a couple of years, the, the White House announcement the OS PT, which is I guess the, the policy group, or will require by the end of 2015 they will require data sharing alongside any publication. So, I think there's a number of different players in place and I, I hope some of those comments are helpful. Yeah, thank you. Well, David pointed out a lot of good things like informed consent has to be stated differently to begin with for clinical data. I also want to point out the NIH mandated all the grant application going in from January 2023 has to have data management and sharing plan in place, and there has been a lot of guides already out. Some of them involves like things like Mandalay data or some other clinical trial will require the depositing data to NIH data archives and short NDA. So there are a lot of requirements going. There are a lot of requirements at the funding agency level as well as in the journal level, but I think what I need is more incentives. So if there are more incentives, even with that, it's not required people will follow. But right now, there isn't a lot of incentive. Other than they say, I'm having an honorary badge of doing open research practice. There's a very little incentives. Yeah, I'm kind of along these lines of question. I have, you know, obviously to do any sort of research synthesis, you need available data, right? And hopefully that that is open data. A lot of it isn't. And as a research synthesis myself, I often, you know, complain about primary study authors who don't provide data openly. But I've also found that other research synthesis are maybe not the best folks to be talking about the need for open data because they're not doing such a great job of being an open synthesis. And so from that perspective, are there are there specific things that need to be in place for the folks that are doing the synthesis level work to make that open distinct from maybe the primary study data. I'll jump in. All right, I'll go first. All right. Yeah, I think you're raising there's two, there's two things that that are important in here. One is right, like what can and should the synthesis community do to help incentivize and help promote broader open access to everything, not just the data, right? Like, like, we know that's just the tip of the iceberg is way much behind it. And then there's the flip which is as somebody who's doing it. How do you also I think embody those principles yourself, right? And I think that's that that one, the to go hand in hand is just a question of like, who's the community really when you think about it. You know, there's a Roger Pang kind of written an early article about this thinking about computational reproducibility for those that aren't familiar with it. And, you know, there's a, if you ever kind of look at some of the ways I talked about it I subscribed to this which is remember we're talking kind of in a different way where we use papers to communicate with each other. I'm going to switch from the beginning of inception all the way out until I get to the paper, but the reader has to do the opposite they start at my paper and they go all the way backwards. And so our job has got to be to walk them backwards, which is they shouldn't just trust what I say they should be able to show them everything that I'm doing so that means all my decisions of where I get access to information is incredibly important. So for example, right, bibliographic data is a really important thing to pull on depending on what type of synthesis research you're doing. A lot of those are closed open Alex for those aren't familiar with it is a great open source alternative at bibliographic data right so finding ways to help promote and use those resources. Also as a way to kind of help kind of hit it both ways right you're demonstrating the need, hey I need to do this you're making it openly accessible for whoever needs it is open source you don't need to have to pay for this or be it an R one application to get access. And I'm trying hard to document my process so a lot of these protocol methodologies are using registration, pre registration concepts to kind of mark what you've been doing the whole way so that way it's not just a paper at the end and open data. It's open process, and trying to use open tools and open data sets wherever possible. And that's the part I didn't body and that conversation that idea of we're publishing papers or publishing positions will say, and I have to like have this conversation go backwards in time. I think that's the thing to really be in places can I do that and how do I do that well. Yeah, my point was a bit of a more smaller detailed one. For me, it's often also a question of resources. I am working on much smaller scale open science projects and so I totally agree about if there's more incentives from the top for instance from journals etc then resources will follow right so this is really, I think where it needs to start. But given that right now there's not so many resources. I often perceive that that needs to be some leeway to compromise for instance if we talk about open synthesis and if I talk about baby video data. I can't just put them out on the internet right but still so for instance what we're trying to do in another project where we assemble a big corpus of online video data we try to think through, for instance different levels of privacy so this can be shared with other projects much more likely that parents concentrate on also ethically better because babies can't give consent by definition. And so on the one way hand we need more flexibility in what we think about when it's open synthesis and on the other hand, I guess, creative ways and maybe like technological tools that can help us with that for instance we now work with face recognition and background anonymization to make these data better for open synthesis right so I guess these are kind of ways that can also help us to have synthesis occur maybe in a bit different ways than than the straightforward way. I just want to jump in and plug one thing that's a really good point and I think we do need to kind of get very clever. There are a couple examples that are interesting that I've definitely paid attention to the Harvard University privacy tools project is a really interesting one of trying to figure out ways of saying hey I can't share the data, but I can share metadata and I can maybe build ways to let you interact with that data in certain manners that won't disclose the privacy or the legality of the data that's hidden behind it. So I think we are absolutely right right we can get a lot more clever at saying yeah, just because I can't share it doesn't mean I can't share everything around it. And so you know this open by default but then close it when it doesn't make sense to me really resonates really well, because we can't just have everything be open and we can't just say oh, well the data can't be open so that's nothing should be open, because that's also wrong. So I think that really to me this is an opportunity to get more clever and find, you know mechanisms that kind of address these types of issues that you're raising. That's so interesting so Harvard data science privacy project. So basically the idea is to disclose different data you have and which one of those you could actually share is that right. Yeah, I wish you could have it in chat yeah so you can just if you actually just Google Harvard University privacy tools project you'll see it there's another one out west to at Berkeley that I think Uber was involved with. But the idea is to create essentially you know protected portal so that the data is housed and that you can still see the metadata and partially interact with it. Again, you just have to put some boundaries on there because you wouldn't want somebody to get outputs that might disclose and so for example you can't get any individual data right. But even the way that you might analyze or summarize the data could accidentally disclose something, but it's really just creating a barrier to say well can't share the data but I can share metadata can share code, I can maybe even let you interact with it. But I just have to create a firewall. So that way, I'm not accidentally disclosing privacy issues I think it's a really interesting way of trying to be more open, allow exploration, but still have a lot of privacy kept back at the data level just because of the needs to do that at the patient level or the legality level. That's a really helpful example. And so I was actually kind of curious from from the example that you gave I was wondering, you know, was that just you kind of sitting down with your team, or did you have to bring in folks to say oh like these are some creative weight like I'm a little bit curious about kind of the resources and the minds you had to get involved to make that happen if you can talk a little bit about that. Yeah, so I think about the different data privacy levels. This is something that actually a team at MIT who developed an online platform for testing babies, which really boomed during COVID as you can imagine, and called look it was already suggesting and developing. And we had to fine tune it because that one was built with a US population in mind so to adapt it to Japan and what parents in Japan and data regulations that Japan might be might be different. So there we went from something that people already had thought through. And I think about the video anonymization that was indeed a postdoc in the team that I work with that suggested it because I think her partners happen to work at a start up that did this kind of anonymization so it's this. Yeah, so I think this was kind of this classical you have someone that happens to know something kind of moment where we brainstormed how we could make these data, because you know we're interested in where babies look in these studies because that's kind of one of our main dependent variables. So anonymization should not do anything about it but right now we're actually now kind of doing validation studies to run automatic gaze classification systems over data with and without anonymized faces to see whether it's actually true that we can use these data. So it actually this synthesis project has become a very exciting project on its own actually. Yeah. Yeah, that's incredible. And so, since we're starting to talk about some interesting and successful, you know open synthesis projects I don't know that I've defined successful at all for anybody. I was curious to know if there's maybe some other examples because really one of the things I wanted to bring with this panel discussion are different ways of thinking about our data for for these kind of unique situations and maybe open up ways for other folks to think about how to bring open synthesis to their own work so I was curious from, you know, other panelists if there are other projects that we could, you know, talk about and share with folks who are watching. Well, I'll jump in real quick just while there's thing to and then I'll pull off the the look at example that you were doing, which is a great group I know Melissa Klein there she used to work at and so it's a pretty cool organization, but something that's also interesting and is to think about, I'm just going to push it really far out which is a lot of the synthesis is looked. We think of systematic reviews met analysis we're looking backwards right we're obviously doing retrospective analyses. You can think about it if one's willing to do this prospectively right say okay well what if I want to tackle something and I think the psychology community is a great example so many babies is a good example of this if you look at many babies it's a consortium of researchers kind of all over the place, some of which are very much focused and I think exactly in your realm should you want to speak more right but this idea of like we want to tackle this question but we know that in order to do that we need to pull our resources together, and maybe purposely embrace the variability that will have by doing it say at different locations or with slight variations in our methodology. So essentially look at it forward instead of just necessarily backwards and I think that's just it's an interesting way to think about synthesis right which is. Oh no it's not just I've got to look behind me and gather the evidence, which has his own challenges it's can I also do it forward sometimes because that there I could actually design it and be more open in my collaboration. Actually purposely break down the barriers that I know exist such as right oh right we're always doing this from high industrialized, you know, populations in Europe and North America let's break that barrier real quick and do something specifically forward. So I just think that's a really interesting way of thinking about it of baking meta analysis and systematic approaches from the get go into large collaborative projects. Yeah, if I just can jump in there because actually so the video product I just talked about is a many babies project it's called many babies at home. So indeed it's about like gathering in this case a lot of baby online data so it's indeed a really great initiative so I think and what makes many babies successful for instance is that it's really built on what our development community needs and wants so basically people suggest projects and then they they need to gather collaborators to actually make it happen so if you would suggest a topic that wouldn't interest a lot of people then it would be hard to get it going so I think it's really important to have projects that people really are interested in topic wise but also can gain something from for instance a paper with a large sample collected etc. And what I wanted to say second thing which I lost now so please go ahead and I'll jump in back later. I want to say that the psychology experiments, redoing it and combining data for open science collaboration. That has been hugely successful, but clinical trial I want to be released clinical trial data is extremely hard to get participants do not easily say yes to, and it's very difficult to sample and survey them so I know that it's a little bit better if we can somehow condense the period that we collect data but realistically it's very hard and also regulations are all different everywhere like country level differences as well as an institution level differences. So, these are a little bit of a barrier there. I think Emily if I if I take your open synthesis and broadly. I think some examples I can come up with or prosper. I think is successful. I think there's well over probably 100 to 200,000 registrations. So there's a certain openness there. I think the agency for healthcare research and quality of the, they have their system matter preview data repository, which I think is another example of people willing to share. And I think another example is the, the clinical trial unit in Oxford. I mean they're world famous for their breast cancer and IPDMAs individual patient data, and they've got agreement among research groups from around the world to share their data and I think that's a very successful example. And I think another example I would use is Prisma 2020, where we now ask specifically authors to tell us a number of things about data sharing, code sharing. And I think the, the last example I have of a successful story is, is the reprise project that Matt Page in Monash is heading up, which is looking at systematic approaches to determining how reliable systematic reviews of interventions are. And I think medicine has been really quite slow to really look at these issues. And I would argue they should have been, you know, very early on because the ramifications in healthcare are quite profound. Thanks, David. You said that was the reprise project. Yes, yes, it's an R-E-P-R-I-S-E. There are now two or three papers published in BMJ, and I know that the protocol is openly available as well. Great. Great. Thank you. So I'm going to move on to the next question because we've talked about some, you know, successful projects. And really what I, what I wanted you all to kind of think about and reflect on or taking a step back, you know, what were maybe some of the key factors that contribute to the success of these projects. So that folks that want to do this work can think about ways that they can enhance these factors to ultimately, you know, have a successful open synthesis project. So I think one factor is really continuity in terms of funding budget infrastructure resources, etc., which maybe for a big project like Prisma or COS is more probably not that easy but more given than for smaller projects where for instance, once the person that started it goes off to another job and it's a small project then how do you actually kind of take it over. So this is something we have for instance with MetaLab. I'm not a postdoc anymore. I don't have so much time. So is there another postdoc that would want to take it over or not? And so it's a little bit dormant right now. I think, and it is hard for instance for, well, I guess it's hard to get open science funding in any case, but especially if the people that work on this synthesis project are not necessarily open science experts, right. So they would need to sell themselves for something that is not their core expertise. I see some knots here. So I think continuity is really, really important. And sometimes maybe hard to achieve if it's smaller initiatives. I'll jump in. I think I'll actually echo what I had said before. I think one thing that works really well is to think about what the end is. And what I mean by that is really thinking hard about how one can make everything as open as possible and to think about, you know, very clear decisions so that way, you know, that storyline is easy to follow from the reader perspective and if it deviates, that's able to be followed for those that want to understand it. So really that that requires just like it doesn't any other research project writers is trying to say okay well it's not about the outcome it's about what and how I want to make that stuff open so these guidelines are incredibly important I think to think about the processes of information where are we store that how we make it accessible how we make it so that it's all in one place so somebody can easily identify, especially since we still use you know papers generally. And to recognize that's also a moment in time so another project I'll just pepper this in here is because it popped up in my head when David was talking is living systematic reviews by the car and community it's really interesting thing to think is that our systematic reviews are met analysis essentially by the time we do them they're out of date, right, or we hope so, if we're moving along fast enough so this idea of like remembering right our own research is versioned. I think it's just as important remember what we write we version our code we version our data, but our own research is his version so the moment in time I do something is really important to remember including that final output. So, I think that would be mine one which is I tend to find that the ones that are really successful tend to actually narrow down and be really focused on the question and think more broadly in terms of access and openness. And I think it would be really good to talk further about the living systematic reviews. It's a really interesting idea. And I think for me, the most important factor to contributing to any success of open synthesis is actually whether we are going to incentivize and reward our researchers. So, you know, in my world, which may be different from everyone else's world on the panel. And every year I, I have a what they call a merit review. And so what they're interested in for me is, okay, David, how many papers have you published, and how many papers have you published in journals with an impact factor above five. And so what I think would be much better would be to say, okay, if you publish the paper, I'm going to give you one point. But if you've published a paper where you have the data open, I'll give you two points. If you do the data open and you've got the code open, and I'll give you three points, we need to start flipping around how we reward researchers with this obsessiveness of, you know, publications. I was obviously, like the rest of the world quite interested in COVID-19. And then I started to see all of these fancy trials on treatment for COVID, which is wonderful. And then I looked at all of the trials. I can't get any of the data. And so if you again go back to that great open, open science collaboration paper, the question is, are those COVID treatment trials reproducible and get the data? Can't have a look. Why are we rewarding somebody for publishing where a patient may want the data shared? You can't get anything. Who's that helping? How is that advancing anything? And I think it's exactly the same for people who are doing synthesis. There's lots of LSRs for COVID-19. Are they sharing their data? Are they sharing their search strategies? I don't know. And I think I'm always talking about data. But what I really mean is the full gamut, data is no use without code. As a matter of fact, there is some evidence that people are more willing to share their code and their data. How useful is that? Come on. I think that's really important. I think what was mentioned earlier, we need funding to enable open synthesis. It doesn't come from the sky. It requires resources, as indicated on the panel today. Some data sharing is complex. And you can't do that if you don't have any resources. And I think journals and funders need to step up to the plate. By and large, they don't to enable open synthesis to really make it happen. Yeah. And it sounds a little bit like we've kind of moved into some of the barriers to open synthesis, right? Because we all agree that this is a really important thing to be doing and to be thinking about and to be planning for, but not everybody's doing it. And in fact, many scientists are not doing it. And I would actually be curious to hear about in terms of infrastructure, what are the barriers to training and mentorship around doing open synthesis as well? But also, you know, I think you all, I'd love to hear from your perspective what are some of the key barriers that we know about, aside from funding, lack of funding, because everybody wants funding. But what are some other key barriers that we know about to doing open synthesis and have there been some creative solutions so that we don't end on a down note? Let me jump in. Since we talked extensively about barriers to data that are present for various reasons. Another thing that I think about the barriers is that data and computing codes are archived at various places with or without maintenance. So there's no good system to search and systematically search and retrieve them systematically. So oftentimes people get data through out of words of mouth. So I think that's potentially a barrier. And in an image data or open science foundation have lots of data at the individual patient level, but the data that are there would not be representative of the entire evidence body of evidence in the field. I think, again, it has to be some good platform where data can be searched systematically, code can be searched systematically, and we can access them based on our specific purpose of our synthesis. And also there's a problem of interoperability. So what they mean by, for example, I call you to disorder or problems may not be the same in another studies. So at the item level and also scalable, there's a problem of, you know, changeable, I guess, you know, definitions and data. I think in general, more education and collaboration opportunities would be helpful so that the importance of open science is shared and also we have a better trust in one another. And also maybe by communicating, maybe we can build a better incentives. For example, we talked about paper, you know, all your career individuals are extremely stressed for publications, right. Similarly, even when somebody develop an art package, I noticed that our package sometimes are not well maintained so it's to work when art is updated. So package doesn't work or packages as a minimum functions without ever like expanding it. So I think this all suggests that there is an incentive problems. The fact that I published a package is important, but it may not be important that I'm not maintaining it. So I think there has to be some better incentives for especially early career individuals to reflect the amount of work they do to make it open science. So I think I will end there. Thank you. I think it's sort of wanting to end on a higher note is, I mean, two examples to me would be the Center for Open Science and the UK reproducibility network. I think they're doing fabulous work to lower barriers and to sort of facilitate openness in the broadest term. I think you asked are there different barriers for different scientific disciplines. Yeah, there definitely are. I would say data sharing is omniapsant in medicine. It's not a culture. It's all about me, me, me. I want to do it all. I'm not going to give anybody anything. We need to really develop strong educational and training. We need we need openness to be part of the researcher performance assessment. And that's happening in some places. Koara is an example in Europe. And I think we see that Dora, the Declaration of Research Assessment is another example of where they're starting to ask people to think outside the box beyond the publication and beyond journal impact factor. Yeah, I think one problem is that already came up is really kind of the knowledge transmission often. So right. So for instance, there might be a student that wants to do open science, but the advisor does not or does not know how to do it. So that is something where with collaborators, we have tried to write tutorial papers both on doing pre-registration and on doing a meta analysis specifically for our field for developmental scientists. For instance, Prisma is wonderful, but a student might get lost if one of the criteria we don't use for developmental science. We don't do RCTs, right? So we write you don't need to use this criteria. Perfect. So this is something where we think we have found some solution. Another problem where I find it harder to see a good solution is the knowledge transfer across different language barriers and cultures. And again, also like the kind of advantage versus disadvantage researcher population. And if we only talk about Japan where I'm working right now, for instance, it is harder for Japanese researchers to access all this information in English, right? There are some open science initiatives and workshops in Japanese, but there's only specific researchers that work on it. I've also given some talks and done it, but of course this is then additional workload in a sense on a few people, the few people that do it in a specific context, right? And yeah, so I think there and the cultural barriers might be even harder because there might be less advices that would promote it. There might be a bit more bad experiences with actually having shared data with actually international collaborators where because of the language barriers Asian authors have ended up not on a prominent position in the paper, although they did a lot. So they might actually already be less prone to trusting the community because they are a little bit outsiders in this weird community. And this is really something where, sorry, this is not a high note, but it is really hard for me to see a solution to it. But I find it really, really very, very important. And even this kind of panel discussion, I mean, this is prerecorded, so it's fine. But if you don't speak fluent English, it's harder for you to chime in. It's harder for you to ask questions. So really this whole knowledge transfer, especially in open science, works so much over Twitter, over sparse information sharing. So I feel like the delay might be bigger than in specific scientific fields where people are actually experts. So even though they might not be so strong in English, they can follow the topic. So, yeah, this is really a hard problem. That is, yeah, I'm going to see if I can take a view of at least how I think about it and maybe have a good end note, which is it does require I think a lot of different actors have to remember like we're in a social system. And I think one is to not forget that that usually there are other individuals that are involved. David, you're the great job of I think listing up all of almost all of them. Right. And the trick is you actually them all to kind of work together. Right. So if we're going to shift, say hiring and tenure practices will the more that those can get aligned with funding and publication practices, the higher chance of success because they're all moving up at the same rate. But I do think it's happening. I think it's unfortunately slower than some of us would want. And I think, you know, my, my end note would be to make sure that we don't want put her, you know, don't take her foot off the gas pedal, but to recognize that like this is a marathon we're running at the same time. And I do think there's evidence that it happens I'll give you like my small anecdote of maybe some it is working I think, which is, let's look at PubMed, which I think people do use as a source as a nice database it holds a lot of nice information. NIH did a pilot on preprints, and now they're extending their pilot and they're making it so anything that's, you know, has direct linkage to NIH funding is now going to be indexed in PubMed. So think about that change, there's a lot of things rapidly one now we're starting to slowly index preprints alongside of papers. Amazing right it's really hard to do otherwise have to go scouring around different preprint servers if I wanted to do that or what would I think is that ignore it that ignored it as a research product. But the second thing it does is it also incentivizes it now it slowly moves up the value chain that is important, I can find it I can use it, and now it's something that starts to get into that reward system so it takes longer than I think we wish and I wish it was like a bit of a golden but that'll be my happy note which is it does happen and I am, you know, watching it slowly it just takes some time but once you start to see things like that PubMed, you know and age policies change it tells me that it's actually slowly slowly working in our favor. Yeah, thank you for that and I do I'm going to say we're going to have to end because we're at our a lot of time but I appreciate that there have been some solutions and also that there are still barriers and I and naming those in this session is really really important because now we we know how to kind of continue to keep pushing forward. So I just want to thank this wonderful panel for sharing your experience and your reflections I really enjoyed the discussion. And thank you all for for listening and and being a part of Esmar com 2023.