 Hi, and welcome back to Esmerconk 2023. This is our second presentation session on searching and record management. As always, you can ask questions via Twitter by following at eshackathon or by using Slack if you've registered for the conference. Presenters will be answering those questions after the session as well. So our first speaker today is Trevor Riley with an introduction to site source, analyzing sources, methods, and search strategies over to you, Trevor. Hey there, welcome to this presentation on the new site source R package. My name is Trevor Riley, and in this presentation, I'm gonna provide an overview of site source's general functionality and the primary use cases for site source. So the package got its start at last year's hackathon. It's really fantastic to be able to share site source for the first time here at Esmerconk. I'd like to thank Neil Hadaway and the Esmerc planning committee for your support was critical in bringing this team together and getting the project off the ground. I also want to make a note that site source is in its final stages of development and testing. We're looking forward to launching version 1.0 here soon, which will include a fully functional shiny app as well. Before I jump into the package, I also want to take a moment to highlight the members of the site source development team, whose hard work and dedication made this possible. Along with myself, we had Kaitlin Hare, Lucas Walbrook, Matthew Granger, Sarah Young, Chris Prichard, and Neil Hadaway. Okay, so at its core site source has two unique functions. First, the package allows for users to add customizable metadata in three fields. We've named them site source, site string, and site label. And in this presentation, I'll refer to this process as tagging. Tagging citations is the first step users take after uploading citations. By tagging a citation record, users can maintain information on a number of variables, which could include the resource which the record was found, the method used to find it, an iteration or variation of a strategy, such as a search string or the progression of a search string, a research phase, and even certain groups of citations, such as benchmarking articles. The second ability relates to how site source deduplicates records. Instead of working like a traditional process where a user selects a single record to be retained, site source actually merges records to create a single primary record. This primary record maintains the tags provided by the user in the three fields. And it's with these two functions that we can start to produce some really great visualizations and analyze our citation data. If you watched any of the Esmar sessions from previous years or even this year, you might recognize the ASIS package developed by Caitlin Hare. Currently a modified version of ASIS is the backbone of the record merging process. All right, so let's take a look at the first example, analyzing resources and methodologies. We'll start by taking a look at the vignette on comparing topic coverage. In this example, we're using site source to review overlap between databases for literature on the harmful effects of gambling addiction. To do this, we've ran a search for the term gambling harm in the title and abstract fields in Lens, Scopus, Criminal Justice, Abstracts, Psychinfo, and Medline. After loading the package, we can tag our citation files with the source names. And once we've run through the deduplication process, we can start to create some plots of our data. This heat map shows the number of records retrieved from each database, as well as the count of overlapping records found between databases. In this example, we can see the scope is the highest number of records on gambling harm and Criminal Justice Abstracts, the least. We can also see out of the 176 articles found at PubMed, 171 of them were found at Scopus. This next heat map uses the same data but provides a breakdown of overlap percents between two sources. As you can see, 97% of PubMed's 176 articles were also found at Scopus. The next plot is the upset plot. The upset plot is one of my personal favorites and it's just another way of looking at that overlap but in a bit more detail. So you can see here, we have our five databases. You can see Scopus here at the bottom with the most number of citations followed by PubMed. And then you can take a look at how these databases overlapped and in what combination. You can see here that Scopus had the highest number of unique citations followed by Lens and Psychinfo. You can also see that there were only six articles that were found in every single database. All right, for this next example, let's say you want to understand not only how each database performed and look at the unique records each source contributed but you'd also like to know how that searching of traditional databases and platform performed versus two other literature gathering methods. Let's say citation chasing forward on your benchmarking articles and then using a new co-citation machine learning AI you fill in the blank discovery tool. So taking those two different methods. For this case, we can take a look at the source analysis across screening phases vignette. And instead of just using one of the customizable fields we're gonna use two this time. We use site source and site label. All right, so you are first going to install and load the package. You are going to point to the direction of where that data is that you want to evaluate and then you are going to start tagging. So you can see these of the file names and then aligns with the source name and then the label here we have a search because those were search results. You get down to this next one, this is final. So it does not have an actual source and it is labeled as a final because these were the final included studies. You can see that TIAB is the title and abstract. Those were the ones that made it past title and abstract. And above those, you can actually see that we have our two methods. Those are also labeled as search. So we're gonna go through the deduplication process and then we're gonna start taking a look at some of these plots. So again, we have that first map plot which shows overlap between the databases. We have the same one as a percent and then we have that upset plot. But I wanna get down to this next one which is again, one of my favorites. So this is where we actually start to see that site label come into play. At the bottom here, it might be a little difficult to see but this is broken up into three plots, bar plots as you can see and we have actually searched, screened and final. So not only are we looking at the databases across the top here, agriscab, you can only green file and then Web of Science here at the end and these two methods. Well, we're seeing how many unique records that method or source brought in and how many duplicate records. So we've found in at least one other resource or method there and you can see then how that progresses through title and abstract screening and for the final included papers. Just below that plot, we do have the citation summary table. We're still working on this. This is one of the things we wanna just make sure is polished and I think this is going to be a really fantastic addition to the package. It's something I'm really looking forward to seeing and thank you to Alison Patel for the inspiration on this and also for helping us in this regard. All right, one last quick example. This is actually a use case that I hadn't even considered at the beginning but made complete sense when I was working on another project. So I'd like to show this one off benchmark testing. So we had a number of different searches and we wanted to be able to take a look at how well we were finding our benchmarking articles and using site source, we were able to actually do multiple iterations of our search string so much faster. So in this case here, you can actually see that search string four and search string six. I know these are hard to see but they are not contributing to finding any of the benchmarking articles themselves. Unlike you can see search string number five has found three, search string two has done three just by themselves, right? But search string four and six while they are finding benchmarking articles, we also have other strings that are finding those same benchmarking articles. So again, a really interesting way to use site source in helping to develop your search strings and strategy. This was also just a whole lot of fun. I think we went through four or five different iterations of this and this is just one of them. I think this may be our second or third. And then also just heading down here and taking a look at this record level table. It's just fantastic. Lucas put this together and it still just blows my mind. I love it. It's very helpful in the benchmark testing. You can play around with it. You can also download it as a CSV. So if you're interested, come jump in this vignette and take a look. A couple more things before I wrap up, I just wanna point out that site source is available via download on GitHub. We will be working to get it up on Cran soon. And we've also put together some discussion boards. We're really hoping that folks engage with us here. We've got the different use cases outlined and vignettes. And then finally, if you're interested in working with us to further develop site source, if you have some interesting use cases in mind, please drop us a line. Please reach out to us on this discussion board. We're really looking forward to working with you. And that's all for now. Thank you so much and happy site sourcing. Thanks so much, Trevor. Up next, Constantinos Buñecas is going to introduce CCAR, a package for assessing primary study overlap across systematic reviews in overviews. Over to you, Constantinos. Hello and welcome to this presentation. My name is Constantinos Buñecas and I am a postdoctoral researcher at Aristotle University of Thessaloniki in Greece. Firstly, I would like to thank ESmart Conference for the chance to present our work. In this presentation, we are going to discuss about an art package called CCAR that provides easy-to-use function for assessing the degree of overlap of primary studies in an overview of reviews. So before I get started, I would like to introduce the other members of the team. Thodoros Diaconidis helped with coding on the packets and designed the sign-up. Ana Mavromanoli and Ana Metina Haydich helped to publish a relative paper with use cases and examples. The evidence-based medicine pyramid, the pyramid on the left side of this slide, consists of primary studies, observational and experimental, which are the sources of information for clinical evidence. Both systematic reviews, which are the first level of evidence synthesis and overview of reviews, which are the second level of evidence synthesis, can be considered as lenses through which evidence from primary studies is viewed. Therefore, the unit of synthesis for an overview of reviews is the review study. One of the key methodological challenges unique to overview of reviews is the management of overlapping data due to the inclusion of the same primary studies in the reviews. The researchers may need to explore the degree of overlap at the overview level and the outcome level. The correct covered area, CCA index, has been used as a quantitative measure of the extent of primary study overlap across the included reviews. In order to calculate the CCA index, we needed to create a matrix which is usually called citation matrix. All functions included in the CCAR packets use the CCA index. There are two versions of the formula, the original CCA index and the adjusted one. If the authors consider structural missingness when they create the citation matrix, the adjusted version is calculated automatically by the functions of the packets. Structural missingness refers to the missing data of the matrix for a logical reason. For example, primary studies were published after the contact of a specific systematic review. Therefore, it was not possible to be included in the review. This is a chronological structural missingness. As we can see in the CCA formula, the N is the sum of ones in the citation matrix are equals to the number of rows of the matrix corresponding to the unique primary studies in the reviews. C is the number of columns of the matrix corresponding to the number of reviews and the X in the adjusted formula is the sum of the X's in the matrix which denote the structural missingness. Additionally, visual methods, such as van and Euler diagrams, upset plots, hit maps, and node link graphs can be used for depicting potential overlap in overview of reviews. For the rest of the presentation, I'm going to focus on the CCAER packets which is available at github.com slash thdacon slash CCAER and it can be downloaded using the install underscore github function from the DevTools packets. First, the functions of the CCAER packets expect the data frame which is the citation matrix and may be a first view of the overlapping reviews. In this matrix, the first row should contain the names of the included reviews and the first column should contain the names of the unique primary studies included in the reviews. The data frame must contain ones and zeros as well as missing values in case of structural missingness. One should be used to indicate if the primary study has been cited, otherwise zero. Of course, any missing value in the citation matrix affects the results. The packets includes three main functions, the CCA function, the CCA underscore table function and the CCA underscore hit map function. The citation matrix, denoted as cm, is the first argument for each function. Additionally, all functions enable the incorporation of structural missingness in the matrix. The first function, CCAER, calculates the overall CCA index for the entire citation matrix as a proportion and the percentage. Larger values indicate greater overlap of primary studies across the reviews. The second function, CCA underscore table, creates a data frame with the per-wise CCA calculations from the citation matrix. The third function, CCA underscore hit map uses the ggplot2r packets generating an upper triangle hit map which shows the CCA calculations as a percentage for all possible pairs of reviews. The gray diagonal tiles in the graph present information about the number of CINDI primary studies as well as the total number primary studies included in each review. There are some advantages of the hit map plot produced by the CCAER packets. The plot is highly customizable as a ggplot object. It uses a sequential continuous color scale. It enables the incorporation of structural missingness in the citation matrix. It presents the single and total number of primary studies included in each review and it is a publication ready plot. Finally, a CINDI app is under development for those users who are unfamiliar with the command line environment. And here are some relative references. You can find more details about the packets in our recently published paper with the title, CCAER, a packets for assessing primary study overlap across systematic reviews in overviews. Thank you very much for your attention. Many thanks indeed, Konstantinos. Next, we have Leonie Twente, who will be introducing her talk entitled, Implementing Text Mining to Optimize Search Term Selection for Systematic Reviews in Language Education, a case study. Welcome to my contribution to the evidence synthesis and meta-analysis in our conference 2023. My name is Leonie Regina-Adventa and I'm a linguist. In this video, I present a case report of using text mining to support search term selection in the context of a systematic review project in the field of language education. I will first skip some background on our project and touch upon the particular challenges that arose during search term selection. Then I will elaborate on how we implemented text mining by using the Litsearcher package developed by Eliza Grains. Now we'll finish with a short discussion of our experience of using Litsearcher in systematic reviews in the educational sciences. The context of my presentation is a systematic review project carried out between 2018 and 2022. And the core team consisted of Marta Hürfler, Tatjana Vasilieva, Tizal Fil and me. But of course, over the years, a number of additional people contributed substantially and they are acknowledged here. To give you some context on the content of our review, it's been shown many times that language and content learning are interdependent. Since the 1990s, there has been a growing interest in an accelerated development of what's commonly called language-sensitive subject teaching approaches. These approaches are a deductive consequence of this knowledge. Since they explicitly integrate language learning objectives with content learning objectives aiming to promote all students' academic outcomes across the curriculum, wide range of approaches have been developed over the years and in very different school settings across the world. Despite this popularity, it's unclear whether they are really more effective compared to non-language-sensitive teaching approaches, primarily because individual study results are mixed and the diversity of approaches makes it hard to maintain an overview about the evidence available. Thus, we were motivated to do a systematic review, to collate, praise and synthesize the quantitative evidence available on the effectiveness of this very diverse set of classroom interventions that fall under our definition of this umbrella term. We followed a systematic review method, workflow adhering to guidelines, including publishing the protocol and the use of the PICAS framework. Of course, the goal of our search strategy was to retrieve all relevant studies, achieving high recall, while achieving as little relevant studies as possible, achieving high precision. However, designing the search strategy was one of the major challenges in our project. In the highly interdisciplinary field of the education sciences, there's no standardized ontology and terminology is rather fuzzy, particularly from our own experience as well as the scoping work we had done on language sensitive subject teaching approaches. We knew that both for German and English, a wide variety of terms exists in the literature used to index such an electronic databases as well. Thus, taking the example of the intervention category of PICAS, many different labels may denote the interventions studied in the research we are interested in finding. In addition, we are aware that we were likely biased towards the part of the literature using a specific set of those terms. Risking selection bias, if we solely relied on our knowledge of the use of terminology in our searches. Thus, the conditions required us to use a method would help counteract biases and optimize the selection of search terms that best represented the literature we wanted to find. We needed to make sure to identify all synonyms in order to avoid not retrieving potential relevant papers, which may not make reference to the terms we identified all the while limiting the amount of the retrieved literature. Now quest to optimize search transaction we found that a lot of the grains have been working on an implementation of an approach optimizing search transaction for systematic reviews in ecology and evolution. And she implemented her solution in the R language publishing a package called literature and this approach reduces biases because it does not rely solely on the researchers grasp of the literature in search term selection. Basically it uses keyword co-occurrence networks to identify search terms that are most relevant or most indicative of the set of articles that uses as a database. For our project we decided to use a multi-step procedure using literature as a complement to the other steps of search term selection. We started by collaborating with experts to define initial search terms and they were used to conduct main searches in a set of databases using a set of articles as input to the literature workflow resulting in an output compliment in the search terms identified in step one and step three we used other relevant papers to extract further keywords resulting in a final database of search terms manually combined into search strategies accustomed for each database that we searched. To go into detail for the second step our implementation of literature you can see that our main searches yielded 2,668 documents for our English databases Scopus and Eric and 58 documents for our German database FIS-Buildung. On the right hand side you can see some terms that were used in order to construct the naive searches. So these documents were imported into literature using the import function. In fact we had to find a workaround for the FIS-Buildung exports because the database export was not compatible with literature. We included both texts from title and abstract as well as the tagged keywords in our network creation. We played around with the various options in literature working curiously to find a suitable cut-off for English and German respectively. This method resulted in 630 English keywords and 53 German raked keywords as well as 38 of the tagged keywords. Here within the German corpus the raking method didn't work as well and so the tagged keywords were much more useful and we looked at them separately. The exported keywords were inspected manually and we selected a subset of them to contribute to our database of keywords replacing less specific keywords identified by the experts in round one or simply adding to the keyword selection. Okay, so what kinds of search terms did we actually identify at each step? I selected here some that are indicative of the kinds of search terms selected for the Python intervention category at each step. So at the first step you can see that we found mainly terminology describing particular interventions such as scaffolding, SIAP, terms of language and reciprocal teaching. Literature then added some more general n-grams such as academic language instruction, language learning while teaching content which seemed to be very representative of the literature we're looking for. The third step also included many more of those n-grams. So what's the takeaway? We found that using text mining in a more general search term selection process helped identify important search terms and reduced risk of bias in searches for systematic reviews in the education sciences. For example, we extended expert define simple search terms through automatically identified n-grams which is known to improve precision. In the test we conducted with the ERIC database we found that indeed for a set of 14 gold standard studies of optimized search strategy, including the identified search terms from literature had better precision and recall compared to a search using only controlled vocabulary. The problems we encountered were twofold. So one, we had a language problem, literature did not work so well for German as it did for English. And one may need adaptation for importing from databases not included in the literature package. So these are discipline specific databases that may differ substantially, but there are workarounds for this. For example, using base or functions to adapt the database structure to the searcher's needs. In some solutions to problems in systematic every methods may indeed hold across disciplines but require some adjustments depending on terminology, infrastructure, et cetera. Thank you for listening and please share your experiences with using technology such as literature in your systematic review projects. Thanks so much, Leonie. Up next, Jakub Ruskovski is going to be introducing the key role of citation chasing in the evidence synthesis on the gastrointestinal symptoms, prevalence in chronic kidney disease, a case study. Hello, my name is Jakub Ruskovski. Today I would like to present our findings on the key role of citation chasing, indeed evidence synthesis on the gastrointestinal symptoms, prevalence in chronic kidney disease. Additionally, especially for all of you who are learning R, I will show a code to perform a basic meta-analysis of prevalence data. Four background, four introductory facts about chronic kidney disease. The chronic kidney disease or CKD is an umbrella term for any abnormalities of kidney structure or function that have implications for health and are present for at least three months. According to the estimated iteration function, CKD is divided into five stages, from G1 to G5. The last one means kidney failure. CKD is a growing health concern affecting millions of people worldwide. Contrary to its name, chronic kidney disease appears to be a systemic disease. It results from the disruption of a number of physiological processes as kidney function declines. Therefore, CKD patients suffer from such conditions as anemia, cardiovascular disease or even bone disorder. As a consequence, CKD patients experience a wide range of symptoms, including gastrointestinal valve. But how much they are overlooked? As always, the answer should be found in a systematic review. The greatest and most up-to-date systematic review from April, 2022 was based on a search of pre-databases and not very specified citation chasing. They provided meta-analyses on the prevalence of constipation, diarrhea and bloating, but practicing picking the utility of such data was limited because on stages of CKD were merged and analyzed together. A half year later, we published a systematic review, fully dedicated to lower GI symptoms and find a lot more data that enable us to take into account the stages of CKD. In this table, I compared findings from the previous systematic review and our work on the right. In the case of abdominal pain, we provided data that were completely missed in the previous systematic review. In the other cases, we analyzed data for thousands of patients more as in the case of bloating. What I love the most is dividing general outcomes into more specific ones such as constipation into the low frequency of bowel movements, heart-stool consistency or self-reported symptom. It is important because each of them can have other causes and also other health implications. So you can see we were able to collect enough data to provide an estimated prevalence for many specific outcomes. In the case of constipation, we included half more patients than the previous systematic reviews. Okay, so how did we succeed and how did we get additional data that had been overlooked in the previous systematic review? The answer lies in data sources. Besides very broad database search and a conventional citation chasing of included studies, we performed citation chasing of papers that introduced questionnaires that are used to assess symptoms in patients. Taken together, forward citation chasing of these articles added data for more than 2,000 patients, nearly 20% of all included studies in our analysis. How did we carry out this? Firstly, we listed questionnaires that ask about symptoms that we were interested in. Secondly, we found articles that introduced these questionnaires and listed their DOIs. Finally, we used the DOIs to search for studies that use these questionnaires in the CKD populations in practice. Just copy and paste the list of DOIs on citation chaser page, click on the load button, and then go to the citation tab. The next step is to search for all citing articles, wait a while and download the file. This file can be important for abstract screen into let's say Ryan, one of the free application for abstract and title screen. Currently, it is recommended to meta-analyze single proportions data such as prevalence using GLMM. It is easy to conduct such analysis with the meta package. It fits the random intercept logistic progression model for all learning are here I show the code that I use to meta-analyze data and to plot data. Although publication bias by its nature is mainly concerned with studies on the effectiveness of interventions, we have, of course, also done such analyzes. Firstly, I plotted a classic panel plot and checked the significance of the asymmetry of the peter's test, p-value was high. However, this paper from 2018 introduced new methods to look for reporting biases. TOI plots, as you can see, and LFK index. They are frequently used in meta-analyses of prevalence data. In this case, LFK index was ever one, so we found a minor positive reporting bias. It means that studies that reported higher prevalence of self-reported constipation were more probable to be published. We use the alt meta package to perform sensitivity analysis with the first function. We tested whether the choice of a link function significantly affects the estimated prevalence of symptoms. And the next function was used because the two-step method to meta-analyze prevalence data is still popular and there is a lot of controversy around the Freeman 2K transformation. That is mostly used to normalize the proportion distribution before pulling. We concluded, as you can see in this table, that the results are generally stable. GLMM with Coaching Link tends to provide a bit lower estimates while the two-step method has a bit higher values. If you are interested in such analysis, I strongly recommend the paper published in 2022 by Leif and Glin, the author of the alt meta package. Finally, I would like to point out the main limitation we faced with. I hope that somebody from the participants of the conference will be interested in working on any of the challenging issues. Firstly, there is a lack of validated filter that can be used in search strategy for studies assessing the prevalence of disease or symptoms. I believe that it is also possible to automate both equation and optimization of search strategy. Secondly, there are no well-established one-step methods to meta-analyze multi-nomial, ordinal data, such as symptoms, severity, or still consistency types. Since I'm not a programmer, I would love to see examples of code with the implementation of such models. Finally, it was really hard for me to find information on how to modify fonts in the charts plotted with the meta package. Eventually, I had to change them with Corel. Moreover, I did not find a well-established method to plot multi-nomial data. Perhaps there are data artists with nice solutions to enhance standard forest plots. And take home as such, citation chasing should not be limited to studies identified through database searches. If you conduct a meta-analysis of prevalence of symptom, perform citation chasing of the questioners that use, that are used to assess the symptom, the prevalence of symptoms can be meta-analyzed very easily with the meta package and there is a need for more research and more working examples in the area of the meta-analysis of multi-nomial and ordinal data. Thank you for your attention. Many thanks indeed, Jakob. Next today, Mark Lajanes will be introducing a study-centric reference manager for research synthesis in R. Hi, my name is Mark Lajanes and I'm an ecologist at the University of South Florida and I'm gonna take a little bit of your time today to talk about this reference managing R package that I tried to crack a few years ago. There's really three things I wanna discuss. The first is what's out there in terms of conventional reference managing tools. Two, talk about how these things don't quite fit for our research synthesis needs. And then finally three, I'll try to wrap up with a few aspirational goals on these tools. Now I'm very much preaching to the choir here but if you've ever taken on a research synthesis project like a systematic review or meta-analysis, I feel like you see yourself as a virtuoso already with bibliographic information. You've gotten your hands dirty many times searching bibliographic databases, pulling the references, sorting, organizing, deduplicating the entire research synthesis pipeline involves keeping track and managing citations, references, studies. Now in terms of what tools are available out there, there are a lot of neat things in terms of research synthesis that we could use. But if you're approaching a research synthesis project for the first time, you may not know what these tools are and you might try to shoehorn a lot of these bibliographic reference managing tools that already exist into your research synthesis project. I remember years ago, right? Men in LA was the hot stuff, the hot kid on the block and it being an absolute headache to try to shoehorn into a research synthesis. Yes, you can annotate, yes you could code but really the UI, the interface is not designed to facilitate that process whatsoever. Zodero is kind of emerging as a fantastic tool for organizing and managing all these references and you could also kind of shoehorn it for many stages involved in the research synthesis project. But my concern with these tools is it actually makes it difficult for us to navigate quickly what we want and it makes it difficult for us to read and understand the citations that we have to continuously evaluate to be included in our research synthesis projects. So here's what a typical bibliographic reference managing software does, right? It allows you to import the searches from databases. It allows you to search all these references, organize them in a nice way. It allows you to export the citations if you're writing a manuscript. It allows you to link up PDFs or the website associated with the study so you could trace back to that original study so you have access to more than just the citation and the metadata of the study but you have the actual copy of the publication. Newer versions of these tools allow you to actually annotate the PDF itself which is eye-opening in terms of keeping track of many decisions that we do or coding criteria that we have to apply for our synthesis projects but for most users that are using this bibliographic software, the goal is to really organize citations and allow for the formatting of these citations to get plugged into a manuscript. The interface of these things are very much managerial, organizational, right? They're tools and meant to sort, organize the bibliographic stuff but between you and me and anyone who's taken a research synthesis project, our relationship with each individual reference is way more intimate than the casual researcher pulling together references for a manuscript. We need to have clear and clean access to each of the references to make them readable, to understand whether or not they need to be included in our projects and to trace to have a tracking system involved in where they kind of fall out in that entire pipeline. Maybe a study does not get included in the final product, in the final report but at what stage did it get dropped? Having an interface that allows you to cleanly navigate that whole process, I think would greatly facilitate our goals and needs into developing clean reports that are readable and transparent all the way through the entire research synthesis pipeline. So years ago, I tried to crack this in R and R is not the cleanest approach to do this kind of stuff. Clearly, if you were taking this seriously, you do it all in JavaScript, I suppose. Shiny is fantastic for this, but again, I feel like you could just skip that whole part and jump into JavaScript to organize the references. My approach was using Tickle2K just because it provides a quickly generated front end for R. And so I'm gonna continue using that GUI package to develop these tools further. The challenge, of course, is that things are kind of not nice looking. You're dealing with a tool that was developed in the late 80s and 90s and there hasn't been much progression in terms of like a beauty and design. But things are emerging that allow us to kind of make things look nice now. So I feel like now is the time to kind of push these things further. But when I first took a crack at it, things were still kind of ugly, but I feel like there's some neat aspects associated with the reference managing tool that sets it apart from these conventional Mandalay Zodero type approaches to managing references. One is rather than just having a tabulated relationship with all these citations, every paper is openly and easily readable with all the toggling and coding criteria associated with it. So the PDFs, the coding decisions, whether at what point in the research synthesis pipeline was it excluded or included, and also links to the endpoints of the synthesis, whether they're effect sizes or just their relationship in terms of where they get aggregated with other studies is all included wrapped up in how we're used to dealing with references, which is essentially just like a bibliographic list at the end of a publication. So the whole UI is designed to make it look like a collection of references from the end of a publication. So what you get is a readable citation and what gets tagged on the citation are all the coding criteria that were used to label the study. And of course you get quick access to the metadata and other effect size data that are associated with the study. Finally, also what's labeled with each individual study is where they belong in that entire research synthesis pipeline and whether or not you can trace back to its origin and any changes and modifications that occurred when we were trying to push it through to create a report. So aspirationally, I wanna put more work into this reference managing tool, but my views on what's something that is that would facilitate a research synthesis project are often bizarre because I get more involved in trying to develop tools rather than trying to complete a project as a whole. So when I tried to record a giant YouTube course of me doing a research synthesis project from A to Z and posted on YouTube, I needed a tool that allowed the viewers to see exactly what I was up to. And so that was my motivation to create this visual UI for research synthesis to organize bibliographic information. I think it's time for me to kind of push it further and release it. And so what I wanna do is just open discussion on like what would be the interesting things that I could include a little bit more than just this list of coding criteria and references. I do not wanna replace Zotero. I do not wanna replace Mendeley. I just wanna create this clean readable interface that allows us to look at each citation and reference and visualize it quickly at what stage it belonged to in the research synthesis pipeline. And hopefully by creating a highly visual tool that maybe is very intuitive, it's easier to achieve our research synthesis goals. So again, I'm at the University of South Florida and I really appreciate your time. And thanks so much, Mark. That concludes our second presentation session. I hope that you enjoyed it. We've had some amazing talks there. I'm really looking forward to seeing you back again at our next session. So enjoy your day for the time being. If you do have any questions for our presenters, do please keep them coming in by Twitter. If you can tag the presenter or tag us, we'll do our best to make sure that those questions get answered. After the event as well. Thanks very much and see you soon.