 Hi, welcome to Esmael Coch 2022 and special session number seven on building an evidence ecosystem for tool design. As always this session is being live streamed to YouTube and the individual presentations have been pre-recorded and published there as well. Subtitles have been verified and can be auto translated for those individual talks. Automatic subtitles will be available shortly for this live stream. If you have any questions for our presenters you can ask them via the presenters individual tweet from the ES hackathon Twitter account and see them on our feed. Presenters have time at the end to answer questions as yeah as allows for time and we'll endeavour to get through everyone's questions after the event. We'd also like to draw your attention to our code of conduct which is available on the Esmael Coch website at esmaelcoch.github.io And now I'd like to hand over to Sarah Young and to start our session. Thanks very much. All right, welcome everyone. Good morning, good afternoon, good evening to all of our viewers. We're really excited to have you with us today and welcome you to this panel presentation and discussion entitled building the evidence synthesis ecosystem through collaborative tool building. So my name is Sarah Young. I'm a social sciences librarian at Carnegie Mellon University in Pittsburgh, Pennsylvania in the U.S. and my co-organizer of this panel is Trevor Riley. Yes, my name is Trevor Riley and I am the head of public services the National Oceanic and Atmospheric Administration Center Library in Silver Spring, U.S. Okay, so as we know the production of evidence synthesis requires collaboration from many stakeholders and as Nakagawa and colleagues noted in their 2020 publication, a new ecosystem for evidence synthesis, building and sustaining an effective evidence synthesis community really relies on improvements in education, communication, practice and methodology. And so the following presentations are really going to explore these ideas and provide some insights into the opportunities and challenges that exist with building usable, sustainable and accessible open source tools and methods for evidence synthesis and really the benefits of multi stakeholder collaboration. So the series of presentations will be followed with a discussion that's going to be led by Alex Banach-Brown of the Berlin Institute of Health at Charité University Medical Hospital and she was also a co-author on the 2020 evidence synthesis ecosystem paper. So our first presentation will be by Sief Waffen-Schmidt and Elke Hausner both at the Institute for Quality and Efficiency in Healthcare in Germany and their talk is entitled Search Strategy Development at a German Health Technology Assessment Agency, our experience with R from an end user perspective. Yeah, hi everyone, Elke and I, we are going to talk about search strategy development. First we're going to give you some background information then we take a look back because we used R from 2008 to 2011 and we are going to talk about our current situation and give an outlook. So our agency is conducting systematic reviews to inform the German healthcare system. On the base of the systematic reviews decisions in the healthcare section in Germany will be done and Elke and I, we are from the Information Management Department at Igwick, that's a short version of the long version Igwick and in this department we have eight information specialists and we are conducting or developing around about 200 searches per year and from 2008 onwards we were looking for support in the development of search strategies and it was because during the first years of Igwick we had a lot of interaction and discussions with the project managers about the searches. So the project managers would say this is a term or these terms they are super important and they have to be in the search strategy and we have the feeling that these terms might not be so important and they would really increase the garbage of the results of the search so but we had no way of, we had no arguments to solving these discussions. So we started to implement text mining because we wanted to have, because we are from an evidence medicine, our work is based on evidence-based medicine so we thought we would like to base our approach on evidence-based facts and our work is based on Jenkins 2004, that is a publication that describes the development of search filters and we did the same. We generated test sets and we based our search strategies on discussion around the test sets and it's a more objective approach and then we also have more way on discussing and making decisions about the search strategy. I would like to give you an example. This is a search we did in 2006 about prostate cancer without text mining and as you could see you don't really have to understand what each of the search lines do and what they include. Just a look on it, you can see it's very complex and complicated and you don't really know where are the different search concepts, what is done and that is also a thing because of course when you add search terms because someone says oh I woke up in the night and there's another search term we have to add then what happens is the search gets really big and the project manager tells you no yet that's not possible. I can't really have a look at 10,000 hits so you have to find a way to reduce the hits and by doing that before that it was really by chance what you're doing and you were not really sure if that is really the good way. We updated the search in 2010 with our new more objective approach and that is what the search looked like at the end after the update. You could see less search terms and it's a more clear search so that is just an example why we did that and what we have been doing. How did we get there? I take a look back so we were looking in 2008 for a tool that supports our work. We were looking for a text mining tool and there were some tools already available the PAP reminder it's still online so maybe you've heard about that but we were looking for a tool which has a little bit more functions and we were having a colleague in our team he was using R for his statistical analysis and he suggested why don't we try out the package TM and we started working with TM. The author is Ingo Finera and also this tool is still available and it's being updated regularly the last time in November 2020. How we used this tool? What did we use it for? We used it to split the text in separate files that was one thing we did. We prepared the references for analysis it was called transformation and then we performed the text analysis itself also we already had the possibility there to compare our references to a population set so references we could compare our references to see which terms are overrepresented. On the next slide you can see a screenshot from TM in R and you see these lines of coding and then on the next slide is basically the output so we used the frequency table you can see on the upper side and starting from that we choose the terms for developing the search strategy. We identified when we use it really some advantages using a package like TM it was very flexible with almost unlimited functions tokenizer, word stemming or this and we also saw that there are many options to further develop packages to use the other packages in R and so on but we did not use it by them because we did not have the knowledge to actually further develop it. We also saw some disadvantages on the other side it was that there was just no graphic user interface like a shiny app and we needed training in our environment and in the TM package and that basically led to some kind of rejection by the information specialists because they did not have this programming skills and that led to the situation that we switched to WordSTAT another tool in 2011 and that's our current situation. We have been working with WordSTAT for the last few years it's also a very established tool and a program it first released in 1998 it's available in version 9 and the developer is Provales Research. It's a content analysis and text mining software and we basically use WordSTAT very similar to TM but it has this user-friendly interface I show you on the next slide. You can see here you have these different options and you just click click click and then you can work with it much easier than going through these programming lines and try to adjust the code. On the other hand the output from WordSTAT as you see here is very similar to the one in TM so it is very yeah we almost use it in the same way so we have the over-represented free text terms and the keywords we have the TF IDF values set value so it's very similar to TM. Yeah so this is the last part the outlook so we were looking for an alternative for quite a long time now because one of the disadvantages of WordSTAT is that it's very expensive and also with each update you have to pay fees again and the other disadvantage is that WordSTAT was not really made for what we are using it for. I think most users use it for qualitative evidence to analyze qualitative evidence so there is no real way to cooperate or to work together with the software or to have different or new functionalities. As I said it's not free of charge and because of that there is no cooperation possible with other users and that is we found out that is really a limitation for us and we have no way of have further developments because the software is exactly like it is and we have no way of changing it. The other thing is that in the last years it is clear that the future lies in a tool that combines several steps together and the WordSTAT would not be able to do that and it has to be integrated into other applications we have so that is a limitation as well and we are hoping for a tool that has these possibilities so our requirements for a new tool would be that it has a graphic user interface so that we actually can use it that there are further functionalities and that we can develop the tool. Another thing that was obvious is and we did not know that 10 years ago but at the moment it's a real big topic is that there should be no barriers for IT security and that is a limitation as well and it should be an open source or it should have an easy way to access so we have the possibilities to exchange with our information retrieval community. Yeah so these are our experiences with our insert strategy development thank you. Great thanks so much Steve Hanokka. All right our next presentation is entitled the value of accessible packages for stakeholders in government and it will be presented by Trevor Riley of the US National Oceanic and Atmospheric Administration Central Library and Kate Schofield of the Center for Public Health and Environmental Assessment at the US Environmental Protection Agency. All right thanks Sarah. Hi everyone thanks for joining us today. As this presentation is pretty brief I'd like to say up front that I'm really looking forward to the conversations that I come out of today's talks. I'm continuously left with the feeling of awe after speaking with members of the evidence synthesis community especially those as part of the evidence synthesis hackathon community and I hope that the insight that folks gain across the various sessions of the conference will contribute to the idea of building a more lively evidence synthesis ecosystem that we're talking about today. I'm also hopeful that folks gain a better understanding of the challenges that different stakeholder groups face. This session was developed to provide some insight into our particular federal research groups and to provide some examples how we're integrating tools into the research process. So at NOAA we have a number of line offices that have the same level of name recognition as the agency itself such as the National Weather Service, National Ocean Service and the National Marine Fishery Service to name a few. And if you're on NOAA's website you'd see a statement that says that the NOAA science stretches from the surface of the sun to the depths of the ocean floor and this is really represented in the variety of requests that our research team receives. We've done literature reviews on everything from space weather topics to benthic invertebrates and beyond that range of research topics that we handle we're also working with clients with a range of backgrounds and responsibilities. So any given day you could be working with a fisheries biologist, an economist, a recovery coordinator, a meteorologist even the director of a national program. So while scientists in the field may be working on topics such as developing best practices and microplastic extraction and bivalves a program manager might be using the literature to help determine funding priorities and how best to allocate time for research vessels to examples of projects that we've worked on. And so beyond that our clients are we're trying to ensure that our clients have the best available science. This is to make decisions. This is a mandate in the Endangered Species Act and so for projects such as this we'll work to provide as thorough of a review as possible with the Full Text Library to literature and in case you're wondering these two species are the common angel shark and the hawksbill sea turtle and another couple of projects we've done. Another example as mentioned here on the slides is that our work is going into building other resources such as the Great Lakes Aquatic non-indigenous species information system for invasive species. This is a really great resource for researchers in the Great Lakes region. But our work also just doesn't stop at the hard sciences. We work with stakeholders who are legal experts. These folks serve as U.S. representatives for intergovernmental working groups such as the Arctic Council. They're also economists and social scientists who use literature to gather a better understanding of the value of NOAA products and services and examine things such as co-governance and co-management models with indigenous peoples. So it really is a lot and due to our small size we have roughly around three to three and a half FTE equivalents you know we're really looking to develop tools and to integrate tools into our process that can help save time. And because of our size and the work and that we're doing to educate our community we haven't been part of a formal evidence synthesis process such as a systematic review at this point. However our group has worked to developing processes and methods that take into consideration the best practices outlined in guidance such as Prisma S and Roses. And we see it as our responsibility to ensure that process is transparent and repeatable. So this diagram is something that our team has is currently working to finalize so still a little in draft form. It's based on the idea of service design and these types of diagrams go by the name of service blueprints. It has a number of uses including helping our clients better understand our processes. In this case it helps to highlight how we've begun to integrate the use of R packages into our work. You can see here at the bottom that the R packages currently stretch across many of our processes including the scoping, search strategy development, string development, the search itself, and metadata cleaning. So again our team is on the smaller side and we don't have anybody with a strong programming background. So while we can play around with R and experiment a bit, for tools that have any real chance to be adopted into our process, a GUI is a must. So you may be asking yourself if our team is reliant on packages out there with a GUI what really is out there that is available? And that's really the point I'd like to get to is that these packages as the develop need to be accessible by groups that don't have that programming capability or that background. And they're also important because first of all procurement and IT review and the federal government is very time intensive. It can take up to a year, sometimes longer to get things through security for good reason, but they're also very costly because it does slow processes down. So I can't stress enough how important that aspect is. The ability to install and run a package the same day you learn about it or better yet use the GUI and actually start playing around with it speeds up the ability for us to adopt and educate our community about evidence synthesis practices. And second, a lot of these tools have been developed to solve a lot of the same issues that we're running into, specifically ASIS by Caitlin Hare, this deduplication package that was developed and presented last year's conference. And then also Litsearcher developed by Eliza Grims who's going to be speaking here soon. And that's a package that we were able to once Eliza had put out the GUI, we were able to integrate that into our standard processes and procedures very quickly. So again, that GUI aspect is very important to us. Okay, so I said this was going to be very brief. Before I hand you off to Kate, I want to share some of the things our teams that my team is looking ahead towards. We really see an increased uptake and demand for evidence synthesis at NOAA and support going hand in hand with that is the increased need on educating the NOAA community on evidence synthesis methods, the value of the products, and then the different use cases for different stakeholders. We're also working to speed up the process of literature organizations to think about that Endangered Species Act, we know Corpus of Literature. So we're working on doing some automated literature classification through some topic modeling, again hoping to use some of the tools that are being in development. We're also building an internal research citation library. It's a repository for all the gray lit just because when you're doing this type of research, especially environmental sciences, there's a lot of gray lit involved. So we're really looking for tools out there that can help clean metadata and make sure that we're not spending hours and hours just doing data entry really, putting a real value to use and being able to tackle more projects. And then finally, there's our package that is being developed as part of the hackathon this week. It's been a dub site source, and that package is going to hopefully be able to track citation origins. And so this is something that might be able to be put to use and analyzing database efficacy and coverage. So as someone who's relatively new to the evidence synthesis world, these past sessions and the tools that have been presented as part of the evidence synthesis hackathon in the past have made a huge impact on my team's process and methods. Show me that there's a lot of folks out there who want to work on and answer the same questions that I have. So I hope that this presentation was for further discussion, and I look forward to seeing what the community does moving forward. And that being said, I'm going to hand you off to Kate. Thank you. Great. Thanks, Trevor. So I'll speak a bit about my experiences with evidence synthesis at another federal agency in the United States, the US Environmental Protection Agency, or EPA. The US EPA's mission is to protect human health and the environment. And it accomplishes this by many means, which are listed here. The ones that are most relevant for what I'll be talking about today are developing and enforcing regulations, studying environmental issues, and publishing information. Different parts of EPA are focused on these different actions. For example, EPA's program offices like the Office of Water are focused on the policy decisions involved in developing and enforcing regulations while EPA's Office of Research and Development, where I'm located, is focused on conducting primary research and evidence synthesis. And ultimately, these three pieces, primary research, evidence synthesis and policy, and thus the different parts of EPA working on these pieces, need to work in an integrated fashion to support and put forth science-based management decisions. Next slide, please. My position is located in EPA's Center for Public Health and Environmental Assessment, or SOFIA, within the Office of Research and Development. SOFIA develops human health and environmental assessments that support EPA's program offices and regions, states, and tribes. And so SOFIA is tasked with providing the science needed to understand how people in nature interact to support assessments and policies that protect human health and ecological integrity. SOFIA's work centers on the evidence synthesis part of those three components that I highlighted on the previous slide. A large part of SOFIA is focused on the human health part of this, particularly in terms of assessing the health hazards of different chemicals through the Integrated Risk Information System, or IRIS program. The IRIS program has a well-established process for identifying and integrating evidence for health outcomes. SOFIA also includes a much smaller group focused on ecology rather than human health. And our group is often tasked with synthesizing evidence across a diverse set of topics focused on physical, chemical, and biological integrity of ecosystems. Some example assessments that our group has been involved with are shown here. All of these were initiated because a program officer region requested state of the science information about a topic of particular concern to them. The scope of these assessments can vary widely in terms of both the complexity of the topic and the approach taken. And so given this, today I'd just like to quickly highlight a couple of broad challenges that our group faces in our evidence synthesis work. Next slide, please. The first is navigating issues of assessment scope and scale. Sometimes we're asked to synthesize evidence related to a very specific question. I think of these assessments as taking a deep dive into a small pool. This might involve examining the relationships between one or a very small number of specific stressors and endpoints, such as the effect of total nitrogen and total phosphorus concentrations on stream algae. These types of questions may be well suited to systematic review, but we're often limited in the time and other resources we have available to conduct even these fairly finely scoped reviews. At other times we're asked to evaluate evidence related to a much broader question, and our approach tends to shift to more of a shallow dive in a very big pool. Instead of looking at just a limited number of relationships, we need to simultaneously evaluate many, many relationships. The conceptual model shown here is one of 13 diagrams that we developed for an assessment of potential mining impacts in Bristol Bay, Alaska. So this might translate into what would be more than 50 systematic reviews, or maybe one really big systematic map. But usually what we're after is an end product somewhere in the middle that's rigorous and feasible given what are often really tight timelines that we're working under, and that provides us with some quantitative information about at least some of the relationships that we're most interested in. Accessible, easy to use tools, the methods that help us plan, adopt, and communicate assessments at both ends of the spectrum and really everywhere in between are incredibly helpful. A well-developed standard portfolio of evidence synthesis options with clearly identified tools and methods that can be used to save time and improve efficiency under each option would allow us to work with our partners from the outset and throughout the evidence synthesis process to identify what's feasible, given different decisions about assessment scope and scale. Next slide please. And that leads me to the second broad challenge, which is navigating issues of process and perhaps most important communication. In our synthesis work, we're working with a diverse set of partners and stakeholders that often includes folks from a variety of organizations, state and federal governments, nonprofit organizations, academia, and other stakeholders, and folks with diverse backgrounds, concerns, experiences, and areas of expertise. Communicating across this diversity of players about both the science issues relevant to the focal topics and the process issues relevant to how and why to conduct an evidence synthesis is incredibly important, but can be challenging given the varied starting points of everyone involved. As evidence synthesizers, we need to know about available tools and methods that can help at each of these process steps, as well as the strengths and weaknesses of those methods and tools. We also need to be able to communicate this information to everyone involved, especially the people requesting the assessment and those who may be using it to inform policy decisions. Increasing awareness and knowledge about evidence synthesis methods and tools among evidence users, that is the people or groups either explicitly or even implicitly requesting evidence synthesis, would help create some shared understanding of what's feasible, improve evidence user's ability to structure their synthesis requests in ways that can better address the key answers they're looking for, and improve evidence synthesizer's ability to help facilitate those discussions. And finally, I'd like to end by stressing the importance of that evidence accessibility. That is making sure that the evidence incorporated into a synthesis is available to other users for other assessments. And Trevor touched on a different aspect of accessibility, which is the issue of accessibility to the tools themselves, which is also an issue. But given the rate at which new information becomes available, and the fact that many of us are looking at the effects of similar stressors or similar interventions or other similar areas of inquiry, it's really valuable if the evidence accumulated in a synthesis is available to others rather than just sitting in files on our desktops. For many of the folks we work with, this is especially important because they often don't have easy access to published papers. And this need for accessibility applies to both the synthetic results of an analysis and the individual bits of evidence that contributed to those synthetic results. This has been one area of focus for us over the past few years, really trying to build capacity to give other users, both within and outside of EPA, access to evidence we gather so that they can then apply it as needed to their own questions. So thank you for your time today. And Trevor and I look forward to answering any questions you might have in the discussion session. All right, thanks, Kate. And our final presentation today will be by Sarah Young from Carnegie Mellon University Libraries and Eliza Graham from the University of Nevada, Reno. Their presentation is titled, Increasing Access to Evidence Synthesis Methods Through Tool Development and Capacity Building. Great, thanks so much, Trevor. So as Trevor mentioned, I'm Sarah Young. I'm a Social Sciences Librarian based at Carnegie Mellon University in Pittsburgh. And my co-presenter is Eliza. Introduce yourself if you'd like. Yeah, I'm Eliza Graham. I'm a postdoc at the University of Nevada, Reno, where I'm working on insect and bird conservation and climate change. Great, thanks, Eliza. So as a librarian, I support quite a few evidence synthesis projects across a lot of disciplines. And today we're going to be talking about increasing access to evidence synthesis methods through tool development and capacity building. So next slide. So tools and methods, of course, can be built and developed for any step of the evidence synthesis process. But this presentation is really going to focus on the information retrieval phase of this kind of work. And information retrieval is really a foundational step that is essential to pretty much any type of evidence synthesis from systematic reviews to evidence and gap maps, scoping reviews, etc. And as a librarian and an information specialist myself, this is also the step that I'm typically most involved in when I support or conduct evidence synthesis projects. Next slide. So effective information retrieval really lays the foundation for all subsequent steps in evidence synthesis. And it's really the search that forms the initial set of studies from which ultimately the included studies in a review are drawn. And so it's really critical, ultimately, to the conclusions and recommendations that result from a review. Next slide. There are a lot of challenges involved in the information retrieval phase, but any searcher and counters and pretty much any evidence synthesis project. And the decisions made around these challenges can really impact review findings and potentially bias the eventual included studies in a review. So for example, we know there's a general lack of standardized terminology in most fields. And so you can't simply rely on searching for one or two commonly used terms, but you really need to include many possible terms to account for the variation in ways researchers in different disciplinary, geographic, or linguistic contexts refer to a concept. Also determining where to run searches, such as which databases, websites, or search engines to use is also important and could, for example, impact the geographic comprehensiveness of a search. Using search terms, not just in English, but other languages can help mitigate the language bias in the published literature. And this can be really challenging if the person designing a search does not have fluency in a language that might be important to a particular review. So how do we potentially address these challenges with innovations and methods in technology? And I'll let Eliza take it from here. Thanks, Sarah. So I think this whole session is kind of fun about tools and methods and what the challenges are. And so we're going to put together this conceptual diagram to ask, you know, whether these challenges require a new method or a tool. And so in this case, the challenge is choosing search terms. And, you know, it's not clear whether there's methods out there for this. So, you know, like Elka described a lot of the work that they did with text mining, but it's not a specific method for choosing search terms, which is part of the challenge. And so when we're thinking about how to best choose search terms for a review, it's really, we need a conceptual solution before we can move on to making methods or code. And so that's where I kind of started from at one point during my dissertation trying to figure out, like, what is the solution to this problem? Like how do we choose search terms in a way that's like quick and objective, reproducible? And so I'm going to just briefly present an example here and to give you a little bit of context so that the words make sense. I'm going to be talking about birds in fragmented forests. So on the left here, this is a larger forest. This is a smaller forest. It's fragmented because they're separated by this probably power line. And this is a bird that would prefer to be in a large forest, not the small forest. And so if I want to find a bunch of studies related to this topic, there's a whole bunch of different terms that I could possibly search for. And so conceptually thinking about this, what you could do is go out, search for some words related to this. So I could say, okay, this is a bird, it's sensitive to the area of a patch. It doesn't like habitat fragmentation. So I could search for those words and get back a bunch of articles that used those terms. But I know that's not all of the terms people could use to describe this topic. And so what I can do then is go through all of the titles and abstracts and keywords for those articles and pull out potentially good search terms. And so that's what's highlighted in yellow here is I could go through patch openness, that might be related, apparent area sensitivity, here we have larger habitat patches, patch size. And so those aren't terms that I searched for initially, but they're related to the topic. And so we kind of need a way to go through all of these articles and pull back what those related terms are. And so that's kind of a conceptual framework. So it's like, okay, this will work, but it's not really a way to implement it. And so with that is the conceptual solution, we still need methods to do this and then code to audit it. And so the method that I decided to use is a keyword co-occurrence network. And so each keyword, which for those things that I highlighted in yellow and the abstract, is one of these circles. And then each line between them is a co-occurrence. So two words appear in the same abstract, they get a line between them, and that means that they co-occurred. And then we can use various ways to find the cutoff and take just these terms in the center of the network that are the most important. So it's like, okay, now we have a conceptual solution, a method, and there's a whole bunch of really, really messy code that I wrote that's probably somewhere in a GitHub archive that isn't really easy for anyone but me to use, because you kind of have to read my mind that I didn't put comments in it, and none of the names of the functions make sense. And so it's not really that easy for other people to access this, even though like, okay, guys, I have a solution. And so that's kind of the next step in making evidence synthesis methods easier to access is turning them into an R package or a different type of tool in a different language. So maybe the R package split searcher, I'm not going to go into all the details, but essentially it's an R package that implements that solution for finding search terms. And the way it works is it does the whole thing that I just described with the conceptual model so you can import results, and then remove the duplicate articles using the R package synthesizer. And then it pulls out all those terms and suggests them. So based on the methods that it uses, it says, you know, here's some possible terms you can consider. And then you can go through and manually group those. So for the first one, three, I was like, okay, yep, that's a bird, that's a bird, that's a bird. This one, no, I'm not going to use it, it's too broad, and so on. And so you get back all of these terms, and then you can feed them back into a searcher, and it will write a full Boolean search. Because deciding where to truncate, when to truncate, what to put in quotes that can take a while and any typos lead to the same problems. And so it will be that, and I don't expect you to be able to read that. And then as Sarah was mentioning, it's really hard to choose languages to search in. So it also will suggest languages based on how many articles or how many journals actually are published in languages other than English for that topic. And so here it turns out that, you know, Dutch was very relevant. So I had to translate the search into Dutch. I don't know if these are the best terms to use, but it'll at least help me pull back some relevant articles. And so if you want to learn all about the actual methods underlying it, not just the overview, I'm going to read about it in methods in ecology and evolution. And so, you know, at that point as a developer, I'm like, okay, that's done, I made a package, everything's great. But that doesn't necessarily mean that end users can actually access the methods implemented in that. So I'm going to turn it back to Sarah to talk a little bit about that. Great. So one of the barriers to using these tools, and I think we've heard this in our other two presentations as well, is just a lack of specialized skills, largely coding skills. And there really is a quite a steep learning curve here when it comes to learning how to code, both in terms of coding languages, and also the associated software involved. And to some, this can really feel like an insurmountable barrier, especially with those folks with no coding experience, or maybe minimal bandwidth, or resources available to learn, or people working in an environment where there's few people around that have or need this skill. Next slide. So how do we address the skill gap? We know that just learning some basic coding can really empower end users, like librarians and researchers, to use these many open source tools that are being developed for evidence synthesis, like Litsearcher. And while there's many free video tutorials, learning platforms out there, this sort of asynchronous, self-motivated learning isn't necessarily right for everyone. Instead, learning with others in sort of a synchronous live or online environment can really be more effective in some cases. And especially if that learning environment is supportive, if it's practice oriented, and if it offers immediate applications to one's work, this can really facilitate long term retention of skills and knowledge. Next slide. So this is where the Carpentries comes in. The Carpentries is a nonprofit organization that aims to build global capacity for coding and data science skills. Carpentries really focuses on reproducibility and open source tools. And all Carpentries curriculum is built and taught by the community and is made freely available online. Carpentries project is comprised of data Carpentries software Carpentry and library Carpentry. And together they provide really hundreds of hours of curriculum to teach our Python and other open source data science tools and methods. Next slide. So Litsearcher really clearly presents a great opportunity to improve search efficiency and minimize term selection bias, both of which are concerns for very busy information specialists and librarians like myself, who support evidence work in many different contexts as well as researchers doing this kind of work. But probably most librarians don't have coding skills. And so that can present a real barrier to using something like Litsearcher. With library Carpentry in particular, we saw an opportunity to build coding capacity and knowledge amongst this key stakeholder group and evidence synthesis work by developing a Carpentries lesson and introductory R in an evidence synthesis context using Litsearcher. And the credit for this idea really goes to Amelia Calleher. She's a social sciences librarian at Cornell University. This was really her idea. Brad Eliza and myself together as well. And she developed a lot of the content as well. Next slide. So this is probably I think the first evidence synthesis related Carpentries less than out there. And it was piloted to a group of virtual learners in August 2020. And it provides learners with an immediate application of coding to their work conducting and supporting evidence synthesis. Live coding, hands-on exercises and a well-supported learning environment are all hallmarks of a Carpentries workshop. Next slide. So the lesson is now freely available. It's up in the Carpentries incubator, and it's open for anyone to contribute and help build on. There's really a lot of opportunity, I think, with Carpentries to build other lessons related to evidence synthesis using, you know, many of the packages that come out of, for example, the hackathon. And I really encourage folks to learn more about the organization and consider contributing. Eliza. Yeah, so, you know, that's one really good way to make our packages easier for people to access is to actually provide training in R. But then there's one further step to making things easier to access, which have both Elkka and Zeep and Trevor all touched on, which is to make a GUI, so a graphical user interface that I use is just called a click-in point. So you can just click in point and not have to do any coding. And so it's really easy to do this for our packages using Shiny apps, because you can just write all of the code in R and then output a graphical interface. And so I did this for Litsearcher because I kept thinking, oh, you know, it's great if out there it's in our package, but there's so many people who can't actually get into R. And so it does everything that Litsearcher does, but you just can do it by clicking and pointing. So in Litsearcher, if you're using it as a package, you type import results, here you can just browse and upload a file, so you can upload a bunch of bibliographic data, you can remove duplicates just by clicking. And then it'll extract keywords using the exact same logic, spits out all the potential terms that you can see here. And then you can look at the network, not the most useful part of the Dewey, but it shows you kind of like what's going on behind the scenes. And then it'll suggest search terms based on how important they are in the network. And there's all sorts of options to fiddle with this and decide, you know, how important should they be before you'll consider them. And then in addition to just selecting search terms, it lets users write Boolean searches by uploading everything, translating it, doing all that. And then, whoops, sorry, also checking the comprehensiveness of the search. So is it retrieving all the articles that you think it should be retrieving? And so, you know, going from this challenge of how do we actually find search terms through a conceptual solution to a method to a package to a click and point, you know, it's all just making it easier for people to access these tools and making it easier to do really quick evidence synthesis. I'm going to turn it back to Sarah. Great. Thanks, Eliza. So yeah, just to basically wrap up, we talked about identifying challenges and limitations and determining whether a new tool or method is needed, or if an existing tool or methods needs to be more accessible. Communication between end users and developers is critical. I think we've heard this from other presenters as well, especially in identifying needs and accessibility barriers. We focused on skill accessibility in this particular talk, but there's really a huge range of accessibility considerations. And Agui isn't necessarily going to address all of those. So certainly keeping other types of accessibility in mind. And ultimately, we need more communication and collaboration between users and developers. And end user needs should be taken into account when really from the early stages of tool development. So we're really looking forward to hearing some of the discussion on these topics about this and really thinking about ways that the sort of collaboration can be facilitated. So we will go ahead and turn it over to Alex for discussion questions. Thanks, everyone. Thank you to Siv, Elka, Trevor, Kate, Eliza and Sarah for wonderful talks. You've really highlighted how you found open source tools, the barriers that have come with adopting or trying to adopt those tools and trying to address some of those barriers. And I think opening this discussion up in a session like this, in a forum like the Esmar Conf group is a really great starting place. So some of the things that you guys have all highlighted, gurus seem to be a favorite. Security and IT procurement are potential barriers from what I've been hearing in your organizations, as well as bringing in a diverse set of stakeholders and review teams. So both communicating about the evidence synthesis methodology, as well as the pros and cons of adopting various tools in the process. But I just wanted to open up the panel discussion or the discussions. We've all identified that we need to communicate better and that we need to get closer to the development community. The development community needs to get closer to the end users. What are some of the, I guess, tools or services that we have? Or what do we need to build or create to make that happen? Perhaps I can go to Sarah first, as you've been working with a group of people already. Sure. Yeah. I mean, I think, you know, honestly, I think conferences and events like this one are actually a really great way to bring these different groups together. I know working in academia and even in a library, I feel pretty sort of siloed from non-academic institutions and even within a library, a little bit siloed from the research departments. And it's been communities like this that have brought me together with folks like Eliza, folks like Trevor. So I think events like this and other sort of opportunities to interact, especially these days, we can do this virtually quite easily. I think we've found that to be a really seamless and easy way to build relationships in communities with folks that we don't often get a chance to interact with. So certainly encourage more events in this story. And you identified carpentries as a great solution. Trevor from a government or a federal perspective is carpentries a type of service or a group that you and your team might be able to engage with in future? I'd like to say yes, because I'm so interested in it. And if I had the time, I would be diving in headfirst, right, into something like that. I just don't have the bandwidth. And my team is so small and we're receiving so many requests that we just can't keep up and do that at the same time. You know, we're developing internal processes to mirror evidence synthesis practices and guidance. And while I want to do that, for me, I really, I think the more beneficial thing would be to work with people who already have that skill and ability. I reached out to Eliza last year, I think before the GUI was already up. And I was just like, so I was able to make it through like the first few lines of code and kind of figure things out. But after that, I'm blind and we're trying to figure this out because it would be a really great addition. And then a few months later, she came out with a GUI. She sent me an email and I was just like, oh, thank goodness. I was like, this is so wonderful. And we've really been able to put it into practice. And it's been an incredible tool. So I see, you know, the group here, Evidence Synthesis Hackathon, you know, being people who I'd like to work more with, I don't know what the format really looks like. But for me, it's just the bandwidth that we don't have to be able to jump into that. So yeah. I feel like, too, it's really important for users to reach out to developers because I didn't know that a GUI was really important until Trevor emailed me. And so like that was the motivation to make a GUI was like, oh, because, you know, I'm in an academic department, everyone uses R for everything here. And so like, I assume that in our package is the solution. And that, you know, it's documented, hasn't been yet, it's all good. And so I think it's important for users to, you know, not limit themselves by what tools are already out there and what they can do. But to email and say like, hey, can you do this? Like, would this be possible? I think it only took me like two days once this hat town to actually work on it. So it's like, you know, it's not really a huge limitation. It just, you sometimes need to give people a little bit of a nudge to do that. Yeah, that last step about bridging the gap between the synthesis teams or the end user and what the tool, I mean, the tool has the capacity, but just that final gap. Steve, did you have your hand up? Did I see? Yeah. Yes, yes, yes. So we try to introduce automation in different kind of levels in our institution or in our department. First, of course, we visited a carpentry in Germany. And also we have one information specialist of these eight who should focus on these topics. And so she's browsing the internet for a new literature or new tools and so on. And she's doing training like the carpentry or just a basic introduction to Python and so on. And also in our healthcare system, there are HTA agencies all over the world. And we have a very, very good network we are working with. And there are a lot of other agencies that are interested in these topics as well. So from this point of view, we have a really good understanding and cooperation. But the actual doing that is a problem. Yeah, yeah. So it sounds almost like I think, yeah, Eliza's conceptual diagram really highlights it really well. There's an idea. There's a conceptual solution. There's a method or a proposed bit of code that then gets put into practice with that last step of having a kind of GUI or some interface or some way for that to engage. But that last bit about having a network surrounding this framework, I guess, about people being able to chip in at various stages because often tool developers, they've got a tool. They don't have access to evidence synthesis researchers or communities. They don't even necessarily know that the code or the solution that they've got can be applied in this situation. So yeah, we can potentially think about ways that we can move towards having this slightly wider community or conceptual framework around how tools get built. And yeah, different ways that tool development can evolve, I guess, naturally because a lot of these tools are built from a need, right? Yeah. Alex, can I chime in with a point? Yes, please Kate, go ahead. I'd also like to stress that the almost synthesis of evidence synthesis tools is really useful. So in our group, most of us are ecologists and we're coming from very topically specific backgrounds. And so a lot of our time has spent more on the trying to understand the particulars of whatever scientific question we're asking. And we're we're stepping our toes into the evidence synthesis methods pieces and tools a bit, but it's not our primary focus. And so having a place where we can really see, okay, these are the tools that are currently available at these different stages. Here's what they can help you do in this process. And really having that spelled out very clearly is really, really helpful. Yeah. And again, that's around communication of the tools and its and its limitations. And I think perhaps, Eliza, you might feel me on this one where you've got a piece of code or a package, but you don't necessarily conceptualize it as being finished already, it's not an end product. You know, sometimes your code or your shiny app is in continuous development. But some way that we can communicate that to users that, you know, this is this is what it does right now. And this is what it could potentially do in future. So that could be a that could be a role for groups like Esmar Conf to provide a communication platform. These hackathons, like the one that Trevor is is is involved with are a really great place to that we can just bridge that final gap between tools and yeah, the end user. I think the the last thing that I wanted to highlight was something that Kate brought up about just that final step that this entire community that we should also take about making the outputs of our it's very meta, the outputs of our evidence synthesis also open and accessible and that we then also communicate how we have developed this, how we've developed our evidence synthesis research products in an open in an open format. Did you have any further thoughts, Kate, on how tool developers could help? Yeah, essentially making that more transparent, more tangible to to perhaps a layperson, how the tools have helped in that process? Yes, so we've done a little bit of work both inside of EPA and then I know other groups have worked on this as well with this idea of sort of crowdsourcing evidence databases that are are putting in the the bits of evidence, not the the synthesis piece or the results of meta analysis but the the bits that are feeding into that meta analysis. We've approached it from the perspective of we work with a lot of states that are looking at stressor response relationships for causes of stream biological impairment. And so they want to know all of the relationships that have been reported in the literature say between nitrogen concentrations and chlorophyll hay or something like that and creating a database of those relationships that multiple people can feed into and build upon what's already been entered by others and then access that information has been really useful and I think is something that that people are interested in. But then there's all the associated difficulties with how do you maintain those kind of databases and touching on what Trevor was talking about about working in the federal government. It's often really difficult to develop these kinds of IT resources and continually keep them funded and maintain them. So partnering with outside organizations on those kinds of things I think is would be really helpful. Yeah and a platform like R which has both user interface capabilities as well as pipeline integration like Sif and Elko were were touching on that a platform like Python or R where you can plug and play different tools would enable a freely available resource like that to be more easily maintained. For the purposes of time I just wanted to hand back to Sarah to close the session. Thanks everyone. Yeah thanks so much Alec. So we're going to move into a live discussion and that will just give us an opportunity hopefully to sort of interact with the folks who are viewing this this panel and just wanted to thank everyone on this panel is really really interesting. It's been wonderful to to work with you all and I look forward to more hopefully collaborations and opportunities to interact and moving forward. So thanks everyone. We'll move into live discussion. Hi everyone. Yeah let's keep the discussion going live. We do have a bit of time for some questions. There's one that's come through on YouTube from Andre Masca Reinhass. It's for Eliza and she was unable to join us live but perhaps Sarah you might be able to help us as you're a part of the Carpentries. Andre says Eliza can you say something about the best ways to choose the starting bibliographic data set to run the analysis on litsearcher? Yeah thanks Andre and Eliza might be able to chime in on Twitter later today but in my experience in sort of setting up that initial set of studies it's really about kind of doing a very targeted search with you know just sort of a cursory but targeted set of keywords and I usually shoot for maybe anywhere from like 50 to 200 results to start with. It's just really about getting kind of a representative set of studies. There's certainly going to be plenty of things that show up that might not be super relevant but as long as you're getting a decent number of relevant studies it's a good starting point to sort of start harvesting those keywords from abstracts and titles. That's been my experience and I think Eliza might have some other input and I'll make sure she sees that message on Twitter. Yeah I'm sure she'll be able to chime in later. Just checking the rest of our tweets and checking. I had a question actually to the panel following on from some of the discussions that we had. How can we incentivize building tools at the start for the community rather than often well in my case we build a tool for the purposes of our own project for a need that we have in our research at the moment and then we kind of move beyond and find out that it's useful a colleague asks us oh can I have the tool things like that. How can we incentivize within our work structures building tools fit for the community right from the very beginning if anyone has any thought. A bit of a meta question. Okay I'll try. I don't think that this is necessary in my experience it's better to start with things you really have you need to do and not to to see the bigger picture because when you most of the time when you see the when you try to see and work on the bigger picture it is something that could not be implemented because you have so many things that that so many barriers and challenges and then in the end it's it's not happening so in my experience it's easier like Eliza told us that she had one task and then she she gets she gets more impressions from other people and can work on on this software I think that is something that is more helpful and successful as the other way around but yeah yeah having having the flexibility for both approaches would be really yeah would be really great. Out of curiosity Sarah do this is changing the subject a bit but do you have any other plans for expanding the carpentries the carpentries and evidence such as some really exciting. Yeah I mean I think there's so much room there to expand on that content so we did run one carpentries training with that curriculum at this point it was all coming up on two years ago and it's now in sort of this alpha mode but we're definitely looking at sort of rerunning that training improving the curriculum and pushing it into the full you know carpentries sort of curriculum set so that's definitely one thing we'd like to do but I think you know as these great tools are developed out of the hackathon it really sets up some really nice potential other ways to incorporate evidence synthesis and these open source tools into these are specifically are training lessons that could be developed in carpentries and again sort of yeah with the bigger picture of thinking about capacity building amongst librarians but also researchers you know who are looking to sort of build their skill sets with coding and their evidence synthesis work so yeah I certainly welcome folks I don't know how prevalent carpentries is in this community but I think it's a great community to get involved with and so I'm certainly you know I certainly welcome folks to contribute to our lesson and think about other ways to build on that. Sarah I have a question now are they are they meant to be online just or is it also planned to do this training on site? Yeah great questions the carpentries definitely wasn't very much an onsite training model until the pandemic hit and I think even in the early stages of the pandemic they were fairly reluctant to move to a virtual environment because it was it was really about these in-person opportunities but they made that transition there's a lot of training for carpentries instructors about how to be effective in a virtual environment and so in my experience that has been a really nice transition it's made that kind of training capacity building much more accessible to you know folks who don't have the ability to maybe travel to an onsite training or they're not at an institution that is a member that offers that sort of thing so I think my understanding and my sort of feeling from the discussions in the carpentries community is that I think virtual training is definitely here to stay probably incorporating back some of that in-person experience as well but yeah I think you know they've definitely been able to make that transition pretty nicely and in my experience that virtual training actually works quite quite well. Great to hear that virtual is a success as well and we've got one last question from Neil thinking about accessibility inclusion and diversity and he asks how do we make sure that tools are designed with broad contexts in mind for example we've got people joining the Esmar company have been supported to hire diesel generators and mobile data how can we think about designing tools for people without constant internet access or other constraints I can think of one tool some of the tools that have been built previously at Esmar comps also run within your own RStudio window so you would be able to then download the packages required in a specific time frame or if you were you know at a library or something and then be able to run it locally yourself I think that's one of Martin Westgate's tools the screening one I the name and the evening at the moment but yeah things like rev tools thank you yeah and things like having packages where there is both cloud as well as local capability would help also with accessibility so just to make sure that we're on time for everything thank you very much to all our presenters in this session session seven on building an evidence ecosystem for tool design and thanks to all who have been watching live and hope we hope those who are watching on catch up enjoy the session as well feel free to ask afterwards on Twitter and our presenters will be looking at Twitter afterwards as well and happy to engage and see you all at the next session session eight on developing synthesis community which starts in 15 minutes at 12 30 GMT thanks everyone goodbye thank you bye