 Hi, I'm Tom Lochtefeld, and I'm going to be talking to you about CISREV.com, a free open access programmable review platform. And CISREV can be really helpful for putting together systematic reviews or for just extracting data from documents. So what is a CISREV? CISREVs are projects on CISREV.com where you upload articles or any kind of digital document. Those could come from a PubMed search, from PDFs, or actually any kind of custom data format you want. So you upload articles to a CISREV. You then define review tasks. These are questions you're going to ask reviewers. Things like, should this article be included or non-cluded? You're creating a screening review. Or you could be asking to extract answers to difficult questions, like, what was the species studied in this paper, or what were the outcomes of this paper? So you collect articles, define review tasks, and then CISREV helps you distribute those review tasks to reviewers whom you invite to your project. So projects can have one reviewer, or they can have hundreds of reviewers. CISREV handles the process of distributing those tasks. And actually, it starts to automate those tasks as well. CISREV has a built-in machine learning platform. And so every time your reviewers are reviewing labels, CISREV is learning how to replicate those tasks. Now finally, at any stage in the review, you can export the results into a spreadsheet format or several other formats as well. CISREV is open access. So you can create what we call a public project on CISREV.com. And these are projects that are accessible to the World Wide Web. If you go to CISREV.com, it's last search, and search for cancer, you'll find a whole bunch of projects that people have done reviewing cancer. This helps to reduce redundancy in this space, and it helps to make your work discoverable. Public projects aren't only discoverable on CISREV.com, but we've worked to make sure that Google and Bing and every search engine can easily find public projects. And so if you go on Google and search for CISREV Eris, you'll be able to find all the reviews done by the Eris Surgical Group on CISREV. If you go on Bing and search for CISREV Gene Hunter, you'll be able to find reviews that we did about identifying genes and text, which I'll come back to in a minute. Of course, once you've found a project, you still have to do something with that project and something we've also introduced in CISREV that's, we think, a kind of new idea in this space is the idea of cloneable projects. And so in this project done by the GASI Group and Tamara Lotfi, who I think is presenting here today, there was a project created to understand migrants and refugees and a corpus of documents they were looking at. Now they wanted to replicate this project across eight different groups. And rather than have to create that project over and over again with the same labels, the GASI Group was able to just clone that project eight times. And what you're seeing here are just bar charts showing you the distribution of answers extracted from documents that they were looking at. You could compare these distributions if you wanted, but the point is that you can go to any project on CISREV, any public project and clone those labels. You can even clone the documents if you want. This makes it much easier to create templates and we hope makes it easier to sort of distribute new ideas within the CISMAC review space. Now I'd also like to talk briefly about how CISREV is programmable. And before I get into that, I want to introduce you to a project we did a little more than a year ago called Gene Hunter. In Gene Hunter we invited 10 reviewers to extract genes from titles and abstracts. Here you can see that a reviewer is extracting Y1 from this article about structure regulators of enhancer promoter loops. So they extract that gene and this process is done. We did it on about 2,000 abstracts and after a little while you collect enough data to build a machine learning algorithm, a name-dending recognition algorithm that can automate this process. And I'll show you how to do that in just a second, but it's worth pointing out that once you have this algorithm, if it's good, then you can scale up this process. So you could, for example, search for all the abstracts on longevity and automate the extraction of genes in those abstracts. If you know anything about the longevity literature, you'll know that these genes are probably the genes you would expect. And if you listen to Cynthia Kenyon's talk, she's a famous researcher in longevity at Colico Labs, you'll see her talking about many of the same genes. And so basically what I'm taking you through is the process of creating a CISRIV, extracting genes from documents in a CISRIV, automating that process and scaling it up to a large number of documents. This same sort of process can be useful in many other domains and for many different kinds of NAD detection, like maybe detecting chemicals, for instance. So how would you do it? Well, you'd have to create a CISRIV review for actually doing the process of extracting genes from literature or whatever entity you have. But then you could go to github.com, PyCISRIV, our Python CISRIV client, and you could import that into a Python project and then extract all of your annotations into a Python object. And we've made that Python object specifically so that it works well with SPACI, which is an NLP package. And here you can see how you would create a SPACI model for identifying genes and text given the gene hunter annotations. And the training process is quite simple as well. So you can notice that there's an NLP object here which can be used now after training to identify genes and text. How would you do that? Here's the training steps again. You could just do doc is equal to NLP, but in a sentence, and then you can display the results. Here you can see us running the algorithm on this sentence. You can see MLH1 being identified in a sentence. Again, this is SPACI. This is using SPACI and PyCISRIV. SPACI is a great package. We don't develop that, SPACI does. And yeah, there's a great combination here. You can go to blog.cisriv.com slash simple NER to see how to do this yourself. Many of you may be R developers. If you work with R, we have a package for that as well called RCISRIV. It's still an active development. So we'd love if you came and worked with us on it and let us know what features you'd like to see in it. But if you wanted to extract all of the answers, all of the information you've extracted in review, it's really just two lines. You have to install the project. You have to import the library as well and get your token. Those steps are pretty simple and they're described on github.com slash CISRIV slash R CISRIV. But then getting your data frame out of that is really, really easy. You just use R CISRIV, get level answers and put in your project identifier and you get a big table out. If you want to work with this, we'd really love to talk to you and you should certainly contact us. We'd love to set up some integrations for CISRIV. So finally, if you're a developer, we really want to build integrations with you. Please contact us. If you're a researcher, we think you should use cisriv.com. It's free and it's only $10 a month for some of our advanced tools. If you're an organization, you should talk to us about institutional license. We're actually already working with the Kennedy Krieger Institute to use CISRIV in the classroom. And if you're an organization outside of education, you should talk to us about managed review. We can help you build these machine learning algorithms. We can help you set up complex review projects. So if any of this is of interest to you, please contact me at tomatandselco.co or just go to cisriv.com. Okay, thank you very much.