 Thank you. Martha, can you do it? It is my pleasure to welcome you to the Webcast on Comprehensive Approaches to Defining Library Value. Then thank you for joining us, a couple of logistical things. Everyone will be muted to cut down on background noise. We do welcome questions. Now you'll have to type your questions and need to answer all the answers that we do not address as well as the ones we address during the Webcast will be distributed to attendees after the Webcast along with the recording that will be available on the ARL YouTube channel. This coordinator at the Association of Research Libraries is going today. I'm going to say a few things about that, but we'll also try to organize another session with him. We have here with us today our main speaker, Bruce Kinma, professor of the School of Information Sciences and the Whitman School of Management of Libraries and more recently on innovation and entrepreneurship. He's bringing a lot of valuable experience in this field. I'm a little bit familiar with the LibValue work and I am a last grant that has been happening for the last three years. We want to understand the concept of value and its various dimensions as it applies to libraries and consider some we are developing to estimate value. We want the audience to become familiar with return on investment methodologies, analytical methodologies basically that help you demonstrate the value of your organization. I've been organizing on the LibValue grant and the already available on the ARL YouTube channel of today's Webcast is defining library value. By that we mean the total organization being viewed as a unity rather than distinct services for, for example, our earlier Webcast starting e-books on library common spaces on undergraduate student success. The distinct element of today's Webcast is the comprehensive finding library value. In that respect, it's actually quite similar to the concept we have to parallel the concept we have in measuring service quality on live call where it looks on the total market and the total market definition of what service quality is. So similarly here you'll hear about how to define a library value for the library organization as a whole. We do have two more Webcasts coming down the pike in June and in August. So I'll talk a lot of words about the grant here from the University of Tennessee and Paula Kaufman from the University of Illinois at Urbana-Champaign. ARL is a key partner in this grant and our main goal is to see how we can take these tools and move them forward beyond the duration of the grant. We do have also key partners in this grant and Syracuse University is one of our key partners through Bruce Kinma. Glovaver collaborations, the GISCOL are working closely with Carol Tenopir at the University of Tennessee in testing different aspects of library value and of course all this work would not have been possible without the support from IMLS, the Institute of Museum and Library Services. ARL has a long-standing interest in developing and for demonstrating library value. Historically, you know the descriptive statistics of how many volumes and expenditures we have as reflected in the ARL statistics or what most academic libraries know as statistics that are academic libraries, the ACL Division of ALA. This is the same survey. ARL has been collaborating with ALA, ACL over many, many years to have the same survey all academic libraries. This is of capturing their role of libraries. But in the last, have moved aggressively towards developing new approaches and the flagship approach has been live call that's a standardized user survey that a robust model of library service quality is viewed by faculty and students and it has been applied to more than 1,300 libraries. Different languages are engaged in approaches that are more internal. A climate call, for example, is a survey that looks into the organizational climate and diversity issues within a library environment and we have also worked in the area of emerging resources and services with digital and NSF grant that looked into digital library service quality and the Mines for Libraries protocol that stands for measuring the impact of networked electronic services. It is a protocol that focuses on the purpose of use and the value people derive from the developing tools, live value, quality, service quality and how do you articulate and capture outcomes and value as experienced by the user. And we've been over the last three years. Now, before we move on into the in-depth approach, determining value and return on investment of your area of services and the response options are yes? It warms the social scientists hardly looking into collecting data, the value of their service. Those of you that don't know may have a PhD in economics, both in the business school and in the school there, I work in the 1990s on the cost of digital libraries or on the cost of library loans. I've done a lot of cost studies in the library community and did some work in early 2000 about the cost of online education, too. I was very pleased that Carol Teneper asked me to be part of the LibQual team or LibVal team. In my role, a number of the webcasts, I'm on board as an economist and look at the comprehensive value to see if we could measure the return on investment of the academic library as a whole. I can't resist, but doing a few slides on the economics of the library and economics 101. And when we look at universities, universities to the typical student actually provide what economists call a private good, which means that in individual exchanges, their tuition, they pay for the service or the product that they get, ultimately a degree, but also four years' worth of education. And that's a private exchange of somebody paying for something that they receive. Academic libraries provide public goods, public goods being an economic term, of a good that when it's provided, it's provided to a number of people at the same time. It has joint consumption properties. And therefore, it's much more difficult to figure out the return on that investment. When you have a private good, you assume for at least the value that they're getting from it. They should be getting more value than what they've paid for that service. But when you have a public good, you have a number of people getting value from the provision of the one service, in this case the academic library. And so you have to use tricky economic techniques to figure out how much value everybody gets from that service and then essentially add up across all the individuals to figure out whether the collective benefit is greater than the cost of the individual good or service. A bit of an economic conversation about the comprehensive value, particularly in the context of Syracuse University and with Don King's work on some other universities, Drexel and Bryant. What's the value then that stresses, you know, what are we talking about here? What are the benefit people individuals get from the academic library? Well, I like to cut that into three categories. There is the economic benefit that an individual gets from using the academic library. I get value from logging into the library resources. I get value from going over to the library and physically using the resources. That's all something that's collected by me privately as do all the other users of patrons of the academic library. There's also the environmental value, though, of the academic library services. And I think that's something that we've ignored over the last 20 years. But academic libraries have provided significant environmental value. And I think to some extent without even realizing it, the fact that we provide electronic services to patrons, the fact that I can log into my academic library's website and receive those services without having to drive over there, without having to expand, means that we are saving the environment, right? These are resources that have less impact and therefore this positive externality on users. I used to say in the 1990s when I talked about my own library that the best thing that the Dean of the Library has done for me is to keep me out of the library. And that was a time when all these electronic resources were becoming available. I could reach it at my desktop. And instead of having to physically drive to the library a couple of times a week to take use of physical resources, I could stay in my office and visit the library every day, actually, but not expend resources to get there. And finally, there's the social value of the academic library. What do I mean by that? Well, you don't have a good university. You don't have a great college without having a great library. The two go together. So you know that you're at a good institution when you have a great academic library. There is a social value to the institution of having that piece of it well-funded, well-recognized, in a central location as a hub for students to go to. It's a source of pride, typically, when high school students are taking on tours of the campus. They always pass by the library. The parents, the students feel better about the fact that they're coming to an institution. That's really a social value. That's not a value of use. That's not an environmental value of providing access to digital resources. In terms of that, you can call it a value of pride. I'm proud of my academic library. And that adds to the overall economic value. How can we estimate? What are the methods used to estimate value? There's a variety of methods that have been used over the years. The basic one is size and assessed value. How much do we have? How big is our collection? How many electronic resources do we have? What do we have in special collections? And what is it worth? So it's sort of the stock value of what we have versus the flow value of the use of what we have. The second way to estimate value that we've looked at is we have statistics on use and turnstile counts and downloads of resources and readership of academic resources at the library. How many times are things being used? And that's a signal of value. It's not exactly value, but it signals to us that what we have purchased is actually being used and people are walking into the building and people are connecting to the website. We must be doing the right thing. The slide is called contingent valuation. And it's actually a method that I use, that Don King uses, over and over again in many of the public library studies that you see out there of the value of the public library. And that's really just a survey of individuals to try and get a dollar figure on what's the value of what they're using the academic library for. We'll go through that in a little bit more detail. And it's tricky to ask the question in a survey, and you have to make certain you ask it in the right way. But it's really a question that comes down to if you didn't have your academic library, how much would you spend to go somewhere? And we'll use that as a proxy. So financial grants awarded. This is a study that was done early on in the value work by Judy Luther looking at the University of Illinois and tying back all the research grants that were collected at the end, that were gotten at the University of Illinois, back to the use of the library by the faculty that were principal investigators on those grants. And saying, hey, we've got a correlation here and if it weren't for the library, we would not have gotten some of these grants. So that's a study to grant. And then finally, there's an economist's favorite tool, correlation and causation, trying to do regression. When we fund our academic library better, do we get better outcomes and can we measure those outcomes? Do we get more grants coming in the door? Do we have a higher retention rate? Do we have a higher graduation rate? And our students getting better grades. Our faculty publishing more papers. So you can try and look at the outcomes that you hope for related back to the funding or the resources that's going into the library and seeing if there's a correlation there to try and get another proxy of the value of our academic library. That's a wonderful study, but there are no shortage. I point to the 07 Really Book by Jennifer Harnes and others that does a great job of summarizing those public library studies. Most of those public library studies use contingent valuation. Use asking patrons how much is the value of this service provided to you. And you can see many of those studies. They're all over the board in terms of the return on investment. Anything from $6, $7, $8 down to $3 or close to $4 on ROI. Meaning for every dollar going in, there's $4 or more of value coming out the other end. There is some work on special libraries in Don King who unfortunately couldn't join us today, but he is known for the individual that has done many of the studies on the library's domain and the value of the specialist. There's been very, very little research, surprisingly, on the value of academic libraries studied by Judy Luther that I pointed to the library at the University of Illinois based on the amount of grants received at the University of Illinois. There's a wonderful study by Megan Oakley, one of my colleagues at Syracuse University. She's been there and really identifying and classifying where we might get value. But again, we haven't done a great job of measuring where we might get value. That's been the principal contribution of the LibValue study is looking at trying to measure all those and the library provides value to the campus. There's a lot of great work by Carol Teneper. At the University of Tennessee, she has all of her research linked there. She's got a great personal webpage and if you have the time, visit her website and look at particularly the value of readership and what faculty and students are doing in terms of accessing the journals. Then there's a people, jointly wrote with Kathleen McClure that is out there as a working paper right now that we hope to get published the results that you're going to see today. There's a lot of value of libraries and these are all sort of from the Economist tool bag of things to remember. It's important to think about estimating value and not use. Now use is not exactly value. A good example is the case of users of late night libraries common space. The library might have a lot more users in the middle of the afternoon but those that come late at night there are fewer people there. You know, they're simply using it at a different hour and sometimes you get the sense that they have a much higher value of that use. They might be the serious researchers, the people that are going there to get some work done versus the people in the afternoon at least in our library, some of the students in the afternoon that use it more as a social experience. So while the use is less late at night the value might actually be more because those individuals have access to it at that time. We all know that e-resources garner a lot more use than physical resources when we move from the physical world of having a journal in print copy and providing that online. It seems like it exploded the number of uses that happen for that resource online. That doesn't mean an explosion in value. That may mean that people are browsing it or just reading a paragraph out of it and may actually have less value in each use. You'll see that when I go through some of the numbers at Syracuse. A second thing to remember is opportunity cost and the importance of our opportunity cost and the decisions that we make in the library. Opportunity cost is a comparison of this or that. If we want to make more room for library comments does that then make last room for books? We've got to look at the value of the library comments which may provide significant value for the users of that space. We must oppose that with some other resources and the value of those other resources. Finally and in some sense most importantly estimating cost and value is difficult and a guesstimate what we show with the return on investment is the data that occurred behind us and the value that individuals an expression in 2010 and 11 of the value that the faculty and the student believed the way. Things are changing so rapidly for them with online resources social media, library comments, et cetera. We have to be forward looking. It's great to have this measure and time of what's the return on investment but we can't forget that we have to be innovative. We have to be forward looking. We have to think about return on investment of taking that leap of faith and providing additional resources and services to our bet. You've already heard some webcasts on and we'll certainly hear more. It was broken down into scholarly reading, teaching and learning, e-books, special collections the value of the library comments and these studies that Don King and I did on the comprehensive value. And we are sorry for those who came later you may not have heard but Don is not able to join us today but I do want to mention a couple of things really. We'll try to organize another occasion and cover his approach on the webcast. And before I go a little bit more into Don's material, Amy do you want to go with our second poll question? Whether your library has ever estimated the cost per user of any library services and the response options are yes we were asked by Kansas administrators to do this. Yes we have done this part of getting pilot management information or no we have not done any class in a moment for results to come in. You would say you cannot why you know and he has devoted a lot of what he has done on that Poxel University primarily and here is an overview of those two settings I mentioned there will be a very extensive report coming out of his work. The couple of elements I want you to keep in mind about Don's approach is he has identified an extensive and comprehensive list of services based on Briand and Drexel Universities a total of 77 services and he has surveyed surveys of the use of and values to faculty and staff and students. Furthermore, articulated that for each one of these services and when we have his report we will have again a chance to look at all these methods that he is developing based on that research. As we move forward and into looking from Don's approach to Bruce's approach we have another poll question for you Amy. The next question your library has ever surveyed users about the value of library services yes we have surveyed faculty yes we have surveyed students yes we have surveyed both faculty and students or no we have not surveyed you. Looks like most libraries have surveyed both 55% surveyed only faculty surveyed only students and 39% have not done any. Works still to be done in many libraries so let's see how we survey. A contingent value survey of the services at the Syracuse University Library I have to say thank god for Suzanne Thornt the dean of the library at Syracuse University it is a leap of faith when a dean says yes they value the library. That is a leap of faith to go with somebody hand in hand and say we are going to figure this out means that you are willing to expose yourself to what your faculty and students think of your library so thanks to Suzanne Thornt we were able to do this and her support surveyed faculty in the fall of 2010 and surveyed students in the spring of 2011 sort of at peak time ask them about a variety of services from the value of those services with an extensive data from these surveys and if you advance the slide one more there may be actually response rates on those. Martha if you try yes so we got a great response rate out of faculty of 42% and a great response rate out of students of 98% in large part I have to say because at the time I was serving as the associate provost at the university people responded to the survey because it was coming from the provost office so there is a lot of data here but for the purpose of this call I am only going to give you a short snippet the sort of punch lines from the major question on contingent valuation so we asked them about all the services they used over the last 30 days to try and get a fresh idea ask them about the most recent service they used recalling the most recent service they used and then ask them if you couldn't have had access to that how much would you have been willing to pay in time and money to get there from somewhere else how much would you have been willing to pay in time and money to get that from somewhere else it was amazing to me the faculty responses and the students responses so in terms of in-person resources looking at the last most recent use of an in-person resource in your mind as a faculty member on average faculty were willing to pay 169 minutes of time in other words getting in their car and driving somewhere to get that resource for somewhere else and an average of $67 for that in-person resource for the remote resources so logging into the library and getting access to that resource faculty would have been willing to pay 94 minutes on average $30 to get that resource from somewhere else it was amazing and overwhelming the response that we got from faculty and students I was very pleased with these responses students, as you might expect willing to pay less they don't have as much money they don't have as much time they feel as though they can get everything they need on the web from Google but students were willing to pay 35 minutes on average and $5 surprising results but very strong results for access to the most recent in-person resource they got from going physically to the academic library and they were willing to pay an average of 32 minutes and $13 for the resource they got online from the academic library so if we can advance the slide I don't know if there are any questions about that but I do want to move us forward to what does that mean for the return on the academic library at Circus University well we take those answers we have to wait the responses to the survey because the survey was slightly different survey responses were slightly different than the overall population at the university so I have to use this statistical technique to wait those responses I have to use another statistical technique to make to average that out over the year because obviously asking people at peak hours during the peak time is not like asking them during the summer so we have to average that out over the course of the year their use of those resources we have to take those values multiply it by the use multiply it by the number of faculty take the values from the students multiply it by their use using turnstile accounts or web logs for student use and for faculty use to get the number of students to get the total value from academic resources used the annual value looking at this table the annual value to faculty at Circus University in time of in-person use is $10.2 million per year back in 2010-11 the total value of in-person use and money is the total value to the faculty at Circus University of the academic library for in-person use of $13.6 million and a total value of remote use to faculty of $19 million per year and if you look at the students those numbers are significantly higher particularly for in-person use but that's only because we have a lot more students Circus University has 21,000 students about a thousand full-time faculty so in total for students the physical library is worth about $23.1 million per year to those students whereas remote access to the library is worth $14.5 million to students per year based on contingent valuation we also did a study of as part of this study how did you get to the academic library how many times do you remotely log in and what if instead of allowing you to remotely log in you have to physically use the library to get an estimate of the environmental impact of that use and then when you physically visit the library how do you visit it do you drive, do you walk, do you bike do you take public transportation and doing calculations on all the ways that you get to the physical library because if I take away remote use from you what are the various ways that you get to the physical library as a faculty member or as a student and so to measure the environmental savings from providing remote access at Circus University back in 2010-11 that value is $1.6 million per year to faculty and $3.7 million per year to students and I should say that value to the environment is $3.7 and $1.6 million and then finally we played the thought experiment too of what if you don't have digital access to anything even going to the physical library no digital access you're going to have to look at these things physically and print them out or print out what you don't already print or potentially photocopy some of the things that you would physically access again a series of questions about your behavior and what if I had to substitute that with physical use and we estimated that the value to faculty is about $100,000 per year and about $700,000 a year for students annually of doing that now we did not estimate the social value I sometimes call that the icing on the cake the value of the academic library is really the use of the library and then we get more value from the environmental use of the library and the icing on the cake is the social value so that's sort of even more so when we add up all those values that we know that the total value maybe $34.3 million to faculty and $42.0 million to students well that would be even greater if we found a way to measure the social value that would be even greater than that all of these estimates I should say are very conservative estimates of value to faculty and students so what does that mean well back in 2010-11 the library at Syracuse had an annual budget of about $17 million when we look at the economic value to adding together what the students say and what the faculty say that's about $70.2 million annually and value and benefit so the return on investment to private benefit to the users would be 4.13 you get that by dividing 70.2 by 17 you get a ROI of 4.13 to the library if you include the environmental value as we've estimated it that's a total of $76.3 million and the ROI is 4.49 and clearly if you were to include the social value if we were to add that in there you would hopefully get an ROI of 5.0 or more by adding that last piece of the coin so that's the comprehensive value to the academic library at Syracuse University using the contingent value method fairly standard and well-known and reputable method out of economics that's particularly used for valuing the environment what's the value of the Yellowstone Park what's the value of clean water used extensively to measure the ROI of public libraries over the last 20 years and now I think for the first time used to measure the value of an academic library with very favorable results very positive very favorable results thank you very much Bruce comprehensive approach is very intriguing and how what's your advice in terms of if a library had to to move forward in this area when would they go for using the comprehensive approach versus when might they want to develop methods or even contingent evaluation of specific services whether it's eBooks common spaces or other more specific elements well I guess the answer to that is somewhat straightforward right in terms of talking to the university to the provost to the chancellor to the president about what does the library do for the institution having a simple straightforward number this is the ROI we've measured it we show this the ROI from Syracuse University and saying hey there's a study out there this is the return on investment that we're getting from our library is helpful in determining the overall library budget but as you're making decisions within the library and I know many academic libraries are facing those decisions of do we want to spend more on our library comments or devote more space to it we're faced with this large price tag for all these online journals and is it really worthwhile to pay for all these online journals or what are our special clocks worth to us it's important to drill down on specific services that's where Don's research is so valuable actually that he provides cost estimates for individual services and that's where the value of many of these other lived value studies are coming in how do we figure out when valuing it or its impact in education what our resources and services in education so that's the time when you're making individual decisions about investment within the library budget of this versus that that's the time I think perhaps to go to the individual studies although I do have to say that those individual studies also provide great ammunition in terms of going to the provost and going to the chancellor and justifying the library's budget you see what our library commons is doing for us do you understand what access to these resources are the impact that's having on our research on campus do you realize that all of these workshops and training programs with the library's offering and how impactful it is in terms of graduation and education on this campus there is an audience question that asks about how this ROI figure compares to what businesses expect out there is there an expectation in terms of what most ROI studies across industries show or in other specific industries I don't think there's an expectation across the board so I'm very familiar with the entrepreneurship in venture capital and angel investment gain and angel investors in venture capitalists always look for that tenfold return they want a tenfold ROI a ROI of ten or more but they want that ROI of ten or more because if they have portfolio of ten investments nine out of ten are failing and so they look for the one that's successful to make up for the the ROI of zero on the other nine in their portfolio that's not the way we can look at a library investment we don't expect that nine out of the ten of investments on the campus are going to fail on us and so we hope to have an ROI of ten on our academic library to make up for the failure with our IT investment or elsewhere these are very strong and very positive ROIs and recognize that an ROI of one means that for every dollar I put in I get a dollar of benefit back so I'm benefit I'm sort of square and that's a very minimum level of which I would make a purchase of a car of a book or anything else for me individually so anything greater than one is good news anything less than one is bad news but anything greater than one is good news and as I said before we've seen this in the public library domain over and over again typically with ROIs of about the same level and it's been very positive news in the public libraries and I think used very successfully in terms of justifying local public library budgets I would like to actually go to one of your earlier slides because I think the next question that's coming relates to to that it's about knowledge or examples of measuring the value of efficiencies or the savings experienced by users through use of the library and if I go back to the slides I'm going to bring up the one where you were discussing the how much the faculty and the students were willing to pay and would you So incorporate it into that so there's two ways to look at the value of a public good one is to try and measure out in the case of the library try and measure out the savings that we might achieve try and measure out the impact it has on retention and then saying while we retain the student therefore they're going to pay a tax amount in tuition and this is going to be meaningful by this amount or try and figure out their increased success in the job market because we have a great library they're getting paid an extra thousand dollars or they wouldn't have gotten that without a great library the reason I like a contingent valuation method is because it's going directly to the user and saying what do you think this is worth to you what is it worth to you and that individual hopefully is incorporating to their decision process the fact that it's saving them time and money and in fact that's the way the survey question is asked if we no longer had an academic library and you were forced to get this from another source one would you have gotten it and two if you were to have it's so important to you that you would have gotten it from somewhere else how much would you have been willing to pay in time and money to do that so asking the individual makes it a very personal decision for them and gets them to think about what is the value to them rather than it being us making a calculation of the time savings the benefits etc etc we are getting directly from the user what they might be willing to spend along those lines another question asked whether the NRI number has been used in any academic library that's led to improved support and budgeting increases and at this point I want to say that one of our co-principal investigators Paula Kaufman who has been leading this effort is the one that says we are not there yet you know I haven't been able to take this figure to my powers and get an increased budget but we are at through the beginnings of understanding these arguments and eventually using them although I do want to mention that there is definitely a lot more focus about delivering value to the students as more and more libraries are introducing library fees so even implicitly the link is there there is one more question have there been any unintended consequences in bringing up how much library services are worth to users any danger of cost shifting from institution to users coming up as a possible solution to funding Chris do you want to respond? I'm not certain I know the answer to that question I mean you know clearly that's a possibility but I think the question is it's still worthwhile to ask what people are getting from their library whether it's to go to a provost and say therefore we need more resources or whether it's to say hey we do need a library fee yeah times are tough and financially things are difficult and you know quite frankly all students and their parents are very upset about the level of tuition so we cannot raise tuition this year but for those services that students use if they recognize that it has value to them then maybe it is willing to we are willing to place a fee on this one of the most valuable resources that we have at Syracuse University and I think some people will laugh at this but it's true is our basketball team and we certainly charge a fee every time people use that when they go to a basketball game should we charge a fee annually imposed on students and the value of the academic library I think at least the students at Syracuse University might not be too upset you are going to get some push back but I would suggest that they find so much value in the resource that they might be willing to accept that it obviously would have to be implemented very carefully as it would on any campus and one last question someone is asking if we know whether there is any differentiation in the value of calculations for undergraduate and graduate students there is a difference so I have all that data too I have it broken down by students in various majors graduate and undergraduate students online students versus campus students so I have the breakdown of all that and it's not surprising to find out that graduate students are willing to pay more than undergraduate students are I should say this too that something I forgot to mention something that Don and I cheerfully disagree on is the fact that in valuing out the time you have to translate that into a dollar figure and I place a value on student time of $10 per hour and showing out the value of time to students the dollar value of time to students of using the library Don thinks that that is way too low of a number that should be a much higher number of time but I think it's important to use the conservative number and see where we're at so my estimates tend to be very conservative and the ROI is very low and very defendable because I use conservative estimates along the way Thank you Thank you very much Bruce it basically means the real value is much larger you mentioned we are not even taking into account the social value component there was actually a person that commented asking if we want if we plan to capture the social value at some point and clearly there is a lot more work to do in all of these areas now many libraries we know look at specific services as Bruce mentioned and as Don's methodology is developing and we have a poll question for you regarding how you regarding specific services Amy has the final poll question asked which one new service the priority for your library to assess either access to e-books a new library common state new services to help faculty teaching and student learning increased access to special collections or none of these but access to other interesting one it says should libraries pursue price sensitivity and awareness do you have any thoughts on that can you repeat that question should libraries pursue price sensitivity price sensitivity analysis I assume and awareness I'm not sure what awareness I'm a big believer in that I harp on it with tuition we have to measure price sensitivity on tuition that will tell us what the future holds and so anybody that providing a service before it should be aware of price sensitivity you know the other as libraries can provide things more efficiently faster should see an increase in use but also should see an increase in value from those services parallel to price sensitivity is time sensitivity of use time is money thank you talks about a lot of good reasons for studying value we do have a website I encourage all of you to go there it captures the rationale behind all these studies faced with difficult economic times and university budget has the value of the library to the wider goals of the university is increasingly questioned so we need good tools now the tools do not always mean just you know whether out there you know there's a whole set of assumptions and approaches Rachel Applegate from Indiana University is coded here having said I hesitate to hang the future of libraries on a demonstrable effect size for the equation L equals success SS so it is not clearly that simple we do need to capture value in a strategic way and I think Bruce mentioned about our ability to innovate and how we can use the value results strategically is a big challenge that's lying ahead of us so it's not only about having the methods and the data but it's also being aware of the questions that are behind the decisions the slide here I recently attended Robert Morrison who wrote a book called analytics at work and he's the awareness of the questions behind decisions at the level of information at the underlying level of insight and there are those questions that are asking about the past so in terms of information you just want to find what happened and report on that on the other hand in terms of the insight that that information can give you regarding the past you're really interested in how and why did something happen why did libraries in the last 100 years have been building the Alexandria ideal of the more is better type of model understand the information you need there you're basically asking what is happening now that information about what's happening now but in terms of the insight you want to have once you have that information what's the next in the future and one information about the future what will happen so we will ask that and we try to extrapolate the underlying we want to gain from that is what's the best to optimize and simulate it's about having good data having good models but ultimately it's about also understanding the and he also had a very encouraging and blissful final saying in his presentation Robert Morrison's presentation in doubt your curiosity influence your friends enable your organization and may your analytics always be good and your judgment even better so data without judgment are good so I hope your judgment is always at least as good as your data and most of the time better than your data hope you will also join us on June 13 for the lead value webcast on success in teaching and research Carol Tan appear will be at that webcast and we want to thank you for attending today's webcast have a great day