 Hello, everyone. My name is Bill Micho from the University of Illinois at Urbana-Champaign, and along with my colleagues, Mary Schlumbach and Elisandro Cabada, welcome you to this presentation on Biblical and Metric and Research Impact Services at the University of Illinois at Urbana-Champaign Library. First, a little background on research impact and research productivity. Universities are in many different contexts utilizing research impact metrics. Most of you have seen them being used, for example, in promotion of tenure. This is something that we do a fair amount of here at Illinois assisting departments with information about research. A number of articles published in Time Sighted H-Index. It's being used in funding decisions, recruiting and hiring decisions, comparative rankings, grant funding applications where they ask for impact indicators, and also just demonstrating the value of a particular university, particular area of university research. The libraries have become much more involved in these type of activities, Biblical and Metric and Research Impact Services. This particular session, the companion presentation is being done by the University of Waterloo people. We both have robust programs with a lot of overlap in terms of our areas of focus. I encourage you to watch their presentation. What we see is a lot of libraries partnering with University administration, colleges, departments in gathering and assessing research metrics. The beauty is that librarians and libraries now have the tools, for example, APIs, Scopus APIs, Web of Science APIs, patent grants, etc. APIs to pull out data. We've got visualization software, things like VossFior and Gefi, data repositories. We have a lot of resources and a lot of expertise to assist in the gathering organization and visualization of research impact metrics. A couple of things really important here, research impact metrics. I've got a quote here about even the most sophisticated metrics are not able to capture the diversity or richness of research impact. There is an increasing interest and concern about what's called responsible metrics. A lot of people have written about this. A number of organizations have been established to look at ways in which we can do more responsible research metrics. A couple of major organizations, the Declaration of Research Assessment, DORA, was established a few years ago to look at how scholarly research is assessed and evaluated. The Leighton Manifesto, I'll link to that here, Leighton University, and they have a Center CWTS for Science and Technology. They have distilled the best practices in metric space research assessment and they're looking at what they call 10 principles for the responsible use of bibliometrics. A couple of other organizations also involved in the science and transition and the end norms, research evaluation and our working group. The questions here and I'm going to show you our system would do a live demo of what are the visualizations that we put together. Questions is how do we actually define research impact? What are the indicators of the metrics that we use in these evaluations? How do we present these in a useful and accurate way? A lot of people are using visualizations and then caveat that we focus on quantitative measures but even the qualitative measures such as peer review and recommendations and other qualitative type measures, they're also subject to bias and error. We've been taking a sort of a matrix view of research indicators and looking at the literature, a view that incorporates a number of factors, a number of indicators. If you look at, for example, articles published by a researcher, in addition just to counts of these articles, you can look at the impact factor of the journals that they're publishing in and we use the Elsevier site score metrics for identifying a number of times articles, articles in a particular journal have been cited over the number of articles published. They're similar to the JCR from my aside journal citation reports. We can look at usage in times we see downloads in a particular journal or a particular article, the altmetric or attention score, looking at the position within the author list, acceptance rate of a journal and, for example, the Cabell publications now give you acceptance rates of particular journals or conferences. So in addition to articles published, the old standby, the gold standard really is a citation analysis, number of times particular articles have been cited. We can also look at the impact of the citing journal, look at citations per year. You'll also look at number of grants received and we've done custom NSF NIH and DOE databases of grants received by the University of Illinois researchers. We can look at the number of patents and we think in many ways the best way to measure innovation is to look at the patents granted, particularly in the engineering and other STEM fields. We use the patents view API and to generate specific custom databases of patents. We've just begun looking at awards and honors. We're now keeping track at the university level here at Illinois of national academy memberships by department. We can look at AAAS. We can look at Nobel prizes, obviously a very prestigious award. Start-up companies, intellectual property revenues, number of co-authors. One of the things we look at for NIH grants is co-authorship within a cohort or group. NIH wants this information provided to them, indicates the amount of collaborative research being done by a particular group or department and then their qualitative measures. Sometimes there's a measure of leadership within the field. Sometimes that's based on social media. Sometimes that's based on appearances in the media. It's often referred to as the voice of the person, peer review, and some other qualitative measures. If you look at the research impact metrics, there's a very rich literature on evaluation and measurement, a number of journals pretty much dedicated just to this topic. Again, much of the focus has been sort of on the gold standard, which is citation data, faculty publications. A lot of the quantitative research metrics have been applied in visualizations and dashboards. A couple of commercial systems, Elsevier, Cyvella Analytics, and ISI Insight. If you look at a couple of some of the research systems, the visualizations are quite complex. This is a system that tries to relate citations with a number of times published and authors in the field. Another very complex system that does a citation analysis and clustering of citations. Our system I'm going to demonstrate here is a dashboard where we try to provide a dynamic and interactive visualization. We've done this over a number of departments and groups, includes a lot of engineering departments, physics, chemistry, particular units such as the cancer center, where we've done a visualization of the research done by individual researchers within that center. We use the Scopus API and supplement that with the Cyvella Experts API to create a bibliographic metadata database, which includes some of the elements that we've talked about earlier, the research impact indicators. This all begins actually with one table with the person's name and Scopus ID numbers. We start building a database from that. A whole series of scripts that we've developed to create a co-authored grant patents received cited by tables. Early last year I did a webinar, Elsevier webinar, which is still viewable on a longer webinar in the system. Other important technical issues here. Our system is a database-driven and it's web-based. Database-driven means you can use pretty much the same scripts for display of information. Just read the information from a database of a particular department. I'm going to show you a display which features display bubbles. All of these are scaled and clickable to get to other information. Again, using the APIs from Scopus and Cyvella Experts in custom databases. Then we're using scalar vector graphics, HTML5, some JavaScript libraries to do some of the displays. The elements that are featured in our dashboard are articles published in a particular time period. The aggregate journal impact total, so we use site score to add up the journal impact for publications by particular researcher. Number of times the article is cited, the NSF NIH or DOE grants, patents received, co-authors, and then a co-author, cohort visualization that we're doing primarily for NIH grants. Let me try to do this demo. This is the bioengineering faculty, the University of Illinois, Nevada, Champaign. There's a fairly small department compared to a lot of the departments here. We are showing articles from 2010 to the present. Each of these bubbles represents a particular impact indicator, so I'll focus on a couple of people here. If I look at Stephen Bowe Park, there's 257 articles from this time period. I clicked here on the 257 bubble. I will actually go, again, these are suspended data taken from a scope of CPI. Articles, the 257 articles he's written in this time period, abstract, a link to the full text, a journal impact site score. I can start these by time-cited, so his high-cited article has been cited 213 times. It's in archipelago transactions of medical engineering. It's an article in Optical Express, IEEE transactions of biomedical engineering. Here's an article in PNAS. It sees the National Academy of Science. You see the site score there is much higher than the site score in some of the IEEE publications. And again, here the links are a lot. This is all totally interactive. A link to PNAS. I grabbed a PDF at this point. I can look at these 89-cited references, and we're getting these right out of scopus. Actually, it's interesting, since I did this database, articles have been cited a number more times. I've been cited 106 times, so a lot of metadata elements here. That's times published. The total journal impact for Stephen Bopart is 868. So these are the 257 articles. The site score added up from those articles. You'll see some of the other people. Rashid Bashir has a much higher impact total, 1189 with 170 articles. You see here, Bopart's article has been cited 356 times. Bashir's 5310 This link here for the times cited takes you into scopus author display with the total number of articles over the course of his career, the total number of citations. Grants. So I click on the 41 grants here from Bopart. It's 11 from NSF and 30 from NIH. I can look at a particular grant. This is a link into the NSF site. It gives more information about that particular grant. Same thing works for NIH. If I go down and NSFs are listed first, the NIH grant clicks into the NIH reporter particular grant project. So group co-authors if I look at the number of co-authors. This tells me that Rohit Bargama has published three articles with Andrew Smith. I click on the three here and get these three articles. All together he's got seven. If I had been Bashir to the center, you can see who he's written with. So this is an indication of the collaborative nature of this particular court. In this case, it's the bioengineering department. It could be within a group, for example, the cancer center. Number of patents works pretty much the same way. I click on the 24 patents here. Actually, it's now 51 patents. It's interesting. And this is a file we updated on a regular basis. Bullport does a lot of work on optical coherence tomography. So you can see fairly easy to do is sort of overarching comparison, eyeball type of comparison. Shumi Ney has published a very few articles or fewer articles than some of the other faculty. What has been cited 53, 5,000 through to 99 times. So this work is just as cited much more frequently. So back to the PowerPoint. This is a picture of the visualization, a visualization on our visual. You see this is a very large a cave of visual that we have in our, what's called our idea lab in the graduate engineering library. We have a version of the visualization that fits to the screen size. So it will fit to the visual or fit to a laptop or actually even fit to a particular phone if that was what you were looking for the visualization. So in summary, kind of think that the research impact services, bibliometric services enhance the role of the library. We're supporting scholarly communication. We're fostering campus partnerships. We use the same scripts and display software for all departments and research groups. We're also looking at the correlations over these research impact indicators. And I've got a couple of slides on that. And we've developed some mechanisms for weighting the indicators and generating composite impact values. So one obvious question is here is the researchers that have published the most articles also have the highest number of items cited. You saw the bioengineering that got some people that published fewer articles, but they've been cited more often. Is there a correlation between the number of articles at the age index, between the number of grants, number of patents? So we did a correlation analysis over 360 faculty. So we combined the departments of Bioengineering, the Cancer Center, Chemistry, Computer Science, Physics, Electrical Computer Engineering, Mechanical Engineering, and then a correlation over these 360 faculty researchers. Looking at correlations between these basically seven elements that we have information on. Articles published, time cited, age index. We code the age index and build it into the database, although we're not displaying it in visualization right now. Site score, total, number of co-authors, etc. So if you look at some of the correlations, for example, between the number of articles, the age index, that's a person's R 0.594. Look at time cited in age index 0.5. So that's relatively low. It actually illustrates some of the known issues with age index, particularly with respect to younger researchers. And if you're only looking at work over the last five to 10 years, you may have a lot of very prolific younger researchers in terms of articles published in time cited, but they may not yet have a big enough age index that correlates highest with that. Whereas you might have some veterans of the faculty who have been in the field for a long time and have high age indexes, but haven't been as productive or prolific over the last few years. In terms of some of the other correlations, if you look at the number of articles versus the time cited, that's very high. So there are some variants across departments, but 0.897, number one is the highest. If you look at the site score and the number of articles to correlation between those two, that's R 0.836. So it's very high. However, if you look at the correlation between the number of articles written in the grants received, or articles written in patents received, those are very low, literally not significant. Even if you look at the patents of the grants, the correlation 0.388 is below a typical threshold value of 0.5. So those results show that the grant and the patents numbers, they actually can provide some useful alternative metrics when compared to articles published at time cited for evaluating research impact and innovation. We're going to expand on this now and add all metric scores. We now have acceptance rates from journalists as part of our Cabell service. We can look at position within the authorship. We can look at major awards and use all those additional factors in some of our visualizations, which actually kind of takes me to the point here. We've developed a system that lets you weight the research indicators. These are the seven that we're actually using here. You're going to assign a particular weight and then come up with a sort of a composite score for a particular faculty member or researcher. Again, this would be something that perhaps the department might want to look at where they might think grants are more important than the number of co-authors or grants are more important than patents or patents are more important than anything. You can use that to put together a composite weighted number for individual researchers. That pretty much concludes my presentation. Thank everybody for listening. If you have any questions, please send an email to my email address. Thank you very much.