 Hello and welcome everyone. My name is Judy Rutenberg and I serve as ARL's Program Director for Transforming Research Libraries. I'm going to introduce our speakers as well as the structure of today's webcast and give you some context for the report and findings to be highlighted here today. Nancy Marin is the Ithaca SNR Program Director for Sustainability and Scholarly Communications and was the lead author of Appraising Our Digital Investment, Sustainability of Digitized Special Collections in ARL Libraries. The full text of that report as well as the survey instrument and data slides are available on the ARL website at www.arl.org as well as Ithaca SNRs at www.sr.ithaca.org. Nancy will speak first, giving an overview of her findings, then she will be followed by three respondents from the ARL Special Collections community. All of our respondents currently serve or recently served on ARL's Transforming Special Collections in the Digital Age Working Group, which played an active advisory role in this project. Our community respondents are Lisa Carter, Associate Director for Special Collections and Area Studies at the Ohio State University. Lisa is also a visiting program officer here at ARL. Ann Kenney, the Carl A. Croft University Librarian at Cornell University and past Chair of ARL Special Collections Working Group, and Ann Thornton, the Andrew W. Mellon Director of the New York Public Library. After Nancy's presentation, each respondent will take a few minutes to react to key facets of the report with both implications and ongoing questions for the Special Collections community. We will conclude the webcast with a Q&A from the audience, saving all questions until the end. For the Q&A, we are joined by Sarah Pickle from Ithaca SNR, Nancy's co-author. When we get to that portion of the program, I will field those questions for our speakers. Appraising our digital investment is the result of a partnership between Ithaca ARL and Ithaca SNR. The project joins Ithaca's expertise and long-standing interest in the area of sustainability with an activity and collection focus critical to ARL libraries, our rare and unique Special Collections. Specifically, the vast collective investment that our membership has made in digitizing those collections and putting them online for a wider, potentially global audience. Welcome, Nancy, Lisa, Ann and Ann. Thank you, Judy. This is Nancy Marin, and I just wanted to thank you, your ARL colleagues and all of today's panelists for joining us to discuss the survey findings today, and also my co-author Sarah Pickle, who's on the line as well. Ithaca SNR is a research and consulting division. We're part of the larger organization Ithaca, which also includes JSTOR and Portico. Over the last five years or so, the team I work with has worked with clients and conducted research into what makes digital projects tick. Our reports, our surveys, and our case studies are all freely available on the Ithaca SNR website, and the link is included here. And this is me. In the survey that we're discussing today, which wasn't developed in a partnership with the team at ARL and guided and tested by experts in the research library community, we're examining the question of sustainability, specifically as it relates to digitized special collections at ARL institutions. So the question, given the investment made in digitizing special collections and given the hope that these can provide an important way for institutions to distinguish themselves, what are the activities that are taking place to support these collections once they are created? Rather than focusing on one single element of sustainability, for example, digital preservation, the survey we all developed here attempted to explore a full suite of activities being undertaken to support the ongoing value of these collections. So to do that, you'll see that we probe several different areas. We looked into the management, we looked into activities and costs, the full suite of activities we could identify. We looked into all the different types of funding sources being pulled together to support these activities. And then we also probed a bit to understand the discovery access and outreach efforts, once it's been created, that are being done to kind of propel the content and the collections into the world. The survey itself had three different parts. The first part was intended to understand the institutional perspective, and this was sent initially to library directors under the signature of ARL's executive director Charles Lowery at the time. After that point, the institutional directors identified the people who would complete the next parts of the survey. They would feed this information to us and we would then directly send out the subsequent sections, the collections in the aggregate to gather information at the aggregate level of an entire library, and then specific collections seeking to find out a bit more information about individual projects. We were thrilled with the response rate. We had a 70% response rate at the top level from the institutions themselves, and then slightly lower as you'll see coming down, but overall a very strong response rate, which ended up being representative of the ARL institutions as a whole. In terms of our key findings, we'll talk about something about the collections characterization first and then into the different sections that I highlighted previously. First, we noticed that most of these respondents had digitized between, let's say, 10 and 40 collections. Some were much higher, and we feel there might have been some questions around how collections or digitized special collections were being defined, but between 10 and 40 was really the sweet spot. The collections involve a tremendous range of content, and here is the first chance to remind you that our full set of data slides is available online, and so this is really going to be just a sample of the information that's in there, and we hope you'll go and consult the final report as well as the data slides to learn a bit more. We noticed that there was a really strong belief across a large segment about the importance of digitized collections to their future. Over 80% of library directors reported that they believed that digitized special collections are critical to their strategic direction. Paired with a second finding, the story changes a little bit, which is that what doesn't change is that it's an interesting piece of information in just the position, which is that there was a sense that the spending in this area, the investment in this area might not be as high as people would like it to be, so about 80% felt there was an important direction, but only about 40% felt that the current investment was meeting what they'd hoped. Overall, the trends reported by most are that spending is going to increase for both upfront and ongoing costs around digitized special collections over the next three years. When we ask further, over 70% reported that these increases are going to be coming from reallocating existing budgets, and then the open question, and what I'm sure that's a question of much debate, is where exactly from where it will be reallocated. We ask directors what their chief concerns were around sustainability, and we've got a tremendous range of answers. There are many concerns around sustainability, but when we coded and sorted these, the top choices were these, and again, there's more data in our report itself. Funding rose to the top. People acknowledge that it's much easier to attract sources and new sources of funding for new work than it is for the ongoing work of sustaining them once they've been created. Storage, preservation, but also other technical questions were a concern. Staffing was a concern both in terms of the time and the types of expertise that directors recognize are going to be necessary in building and also sustaining digital content. And finally, a whole set of issues around establishing and clarifying workflows and integrating these into the business of a library. In terms of how collections, digitized special collections are being managed, you'll get to enjoy a beautiful rainbow of choices, which is what resulted when we asked the question, what is the primary, who has the primary's responsibility, what department does for managing and coordinating the ongoing sustainability of these collections. Now, what we feel from this is that it's clear that there's not really any one place yet. There are a couple of interesting points to point out here. The first is that we were interested in that 19% slice that shows that these are actually new units or new subunits that are being created and designated as devoted to running digital projects. The other thing to point out is that while people responded to the question, several indicated that it's still not that easy to identify a primary site for this work and that it is such a deeply coordinated effort that it's sometimes hard to tease apart who's leading it. So there's obviously a lot of interdepartmental work that goes on in this area and this is something we hope to learn a little bit more about in future phases of this work. We asked who is in charge of managing a specific collection. So what phrase best describes how a collection is managed? And often the answer was that things are being the people managing digitized collections manage several at once. About 20% responded that a collection is run by someone who just manages a single project. This could reflect that there are simply fewer large scale projects out there that require this kind of management. But again, it seems like an interesting point to explore. Here's the fun part. We surveyed people on the expenditures, the sets of activities, and also the costs attached to creating and sustaining digitized special collections. Now this was not easy. We all knew it would not be easy. We worked very closely with partners to develop the easiest set of questions, the most answerable set of questions, but even so, we know that by asking people to break apart activities and attach costs to them does not really correspond to the way that current budgets are managed. That said, we all agreed it would be interesting to find out what can we learn from doing this activity to really understand what those costs are and what the investments are in these different activities. So this is a first taste. Again, there's more data in the report, but here are some top level findings. So for those who were able to report, and the good news is that even with all of the challenges, 80% of our respondents fully answered the section. The upfront costs that libraries are investing far surpass the ongoing costs to maintain them. So if you look at the second bullet here, the range of upfront costs, and we asked people to report on their prior year's investment, between 9,000 and 2 million, so very, very wide range. The ongoing costs from about 100, so some people thought they were investing very little or could break out very little, up to about a million. So a big shift in difference. The upfront costs extremely high, ongoing costs very low. And then forth pointing out that those ongoing costs aren't for one particular project. That's the cost of the activities to support all the activities around all the digitized collections that the library has created since they started creating them. It's also worth pointing out, and again, there's further detail in the report, that the top category shift, and this is something you'd imagine as well. In upfront, you see scanning, project management, and metadata creation right at the top, whereas for ongoing, the highest investments show up in web design and ongoing software development, project management, and preservation. The things you don't see on the list are probably worth mentioning too, and those are activities around editorial development and outreach and things like that. Funding sources. We asked what the different sources of funding were, both for creating work and for sustaining it. You'll see these two charts show a very clear message. The base budget of the library is covering almost everything. Slightly less for upfront work, where external sources are more easily available, but ongoing it's almost entirely the library's budget itself that is covering this work. We asked if institutions had experimented with any form of revenue generation, and about half responded that they had, which led us to actually prompt and find out why the other half hadn't. We found that about half of those who had not tried revenue generation did this for reasons that they felt the activity would be inconsistent with open access mission. We also made very pragmatic choices and said, not that we wouldn't try it, we just don't think it would be worth the investment. The flavor of the feedback is worth sharing too, and again we have some more detail to share in our data. People felt materials needed to be open access, that it would be contrary to our mission, or a donor simply required it to be open access. We also felt it was interesting to take a look here at the broader perspective on open access because fully 60% felt that open access was very, very important, and that includes some folks who actually have tried some revenue generation. When we looked at the types of activity taking place under that umbrella, much of it looks like licensing that at least people have tried and experimented with. But print on demand seems to be a close second, and that might be one of those examples of methods that would be compatible. At the end of the day, though, there's a fairly sobering message, which is that even with the experimentation going on, the total amount being brought in in terms of revenues are quite low. You'll see from this chart where each point represents different institutions, how they reported, almost everything is falling below the $10,000 range in a given year, and most actually fall below four or five. There are a couple of outliers, but they were real outliers. In terms of understanding audience, we asked if institutions were gathering data to understand the composition of their audience. So not just raw usage figures, but really what's in there. About 57% said no, and 43% said yes. When we asked them a little bit more of the 43%, about a quarter of them said that indeed about 25% of their users are coming from what they see as their core audience. What was interesting was about half of them responded that actually most of their users for their online digitized special collections are not from their core audience at all. In terms of the activities people are doing to understand these users, many, many are engaging in statistical analysis using Google Analytics, for example. Very few end up doing more qualitative methods like focus groups. It's also interesting to note, though, that the web analytics tools are sometimes less satisfactory. People find them less effective than they do with focus groups when they can find ways to do them. In terms of discovery, access, and outreach, the key takeaway, there are a few. The first is that there's a wealth of activity going on. Libraries are undertaking a range of activities creating metadata, making content available, optimizing websites for search. There's a tremendous amount of effort to make sure that people can reach the collections once they're there. Again, this is a slide with a lot of data on it, so I hope you'll have a chance to look at it at more leisure on our website. In terms of outreach activities, we also asked about the different things people are doing to reach out to this audience. When we sorted it, we didn't really think about these categories in advance. We developed a tremendous amount of lists of choices for people to respond to in terms of the activities they're doing. What was very interesting is when we did sort it and determine what people were doing most, they seemed to be activities that involved creating a tremendous piece of work, placing it online, the finding aids are tremendously valuable, and those activities, people are doing about 80% of respondents reporting it regularly. There's a drop to the next category that involves more of a direct outreach and interaction with the audience, and that we felt was probably worth pointing out as well. Finally, with all the activity taking place and the wealth of outreach activity taking place, 17% of respondents noted that they are actually measuring the effectiveness of these outreach efforts. So to sum up, on the management question, we see that it's still very dispersed across many departments. For costs, we see that expenditures are very low post-build, and the question there is, is this a sign of an efficient system or a sign that there may be some further investment needed? Funding sources, we see everything is still coming heavily from the core budget. There is a very deep reluctance to experiment with revenue strategies, but a couple of examples of things that may be compatible with open access as well. For understanding audience and outreach, there seem to be few who are closely studying who the users of these online collections are, and in terms of outreach, again, not as much assessment and analysis perhaps of what the impact of some of those efforts are. So some of the big questions going forward. What kinds of collections actually need this level of attention? Maybe some of them do and some of them don't, and how can we think about defining those? For libraries, maybe thinking about what the set of activities are that are currently being undertaken and what areas might be useful to have further investments. And then finally, what might other sources of support be for the activities that you'd like? So I'll end here. Thank you again, everyone, for letting me share these findings with you, and I'm excited as well to hear the comments from experts in the field on how this work might be useful to them. Great. So I think that's over to me. I'm Lisa Carter from Ohio State University Libraries. The first point that resonates with me from this study is that libraries are spending far more to create new resources than they are on maintaining and enhancing the ones that they've already created. Combined with the finding that we do purposely pursue internal and external ways to fund new digitization work, but we do not seem to go out of our way to ensure ongoing support for their management and promotion, this suggests to me that we need to rebalance the investment. In fact, the ongoing support of these collections seems to be absorbed primarily by the library's base budget and seems often to be not very countable. Well, I certainly strongly support dispersed and integrated systems for managing the long-term preservation management and even promotion of our digitized collections. These two findings suggest to me that we need to be looking more carefully at our investment in digitized collections and acknowledge that it's not just all about upfront costs. That acquiring these resources through digitization, grant-funded, or internally project-funded means committing to them for a lifecycle of preservation, migration, updating, and outreach. We need to invest not just in capturing these digital surrogates of our special resources, but also in the longevity and usefulness of those digital objects. Just as we invest in physical storage for our physical collections, pay our utility bills to keep them cool and dry, hire people to retrieve them, put them back when the research is done, employ people to go out into the classroom and community to engage users with our rare items, and offer intensive research services in our reading rooms. We also need to invest in the parallel functions that keep our digitized collections useful, relevant, and accessible. Well, in my mind, while significant accumulations of special collections and their digital surrogates create rich troves of valuable resources to support scholarship, our enthusiasm to acquire and present online materials tends to outpace our resources, infrastructures, and capabilities. If the goal is to enhance research, teaching, and learning with unique resources over a long term, then institutions must regroup and refocus on investing in programmatic strategies and sustainable systems. As we think about more direct and better articulated investment in long-term sustainability of these collections, the other finding that resonates with me is that while the ability to offer greater access emerged as a key motivator for digitizing collections, investments in understanding the needs for the audience are quite low. Libraries say that they digitize to provide greater access, and yet we do not invest time or resources in understanding how our audience uses the digital objects we provide. Well, the first two findings I mentioned are about good stewardship. You know, if you build it, you need to take care of it. This one is about determining return on investment. What is the point of digitizing all these materials and investing in their long-term management if people can't find them, can't use them, or don't know about them? We must make sure our digitized collections are meeting our users' needs. This finding did not surprise me. In my work with the RBMS Task Force on Metrics and Assessment, we are finding that assessment of special collections across the board is a huge challenge. We tend to employ instinctual strategies for articulating the value of our collections and our impressions of the needs of our users. As a profession, we have not found common and satisfactory ways to gather objective evidence of relevance. Our curatorial instinct needs to be confirmed by hard data so that we can articulate the value of the organization's investment in digitization and systems to manage and provide access to digital surrogates long-term. While 43% of libraries in the survey are actively gathering some data about their digital audience, this is often in the form of web analytics and much less often involves what the response described as more effective, qualitative approaches. In addition, while about half of the respondents' report conducting some form of regular outreach, only 17% of respondents say they measured effectiveness. I feel that this is partially because we don't take the time to look back on the impact of our offerings and partially because we don't know how to do that assessment. And yet our parent libraries are increasingly investing in assessment activities, both in the online environment and in areas of instruction and outreach. I feel that we should invite our colleagues in assessment on web development teams and in outreach departments of our libraries and maybe the broader university to help us figure out how to design strategies that help us gain full return on our digital investment. While they don't directly address the assessment of outreach and use of digitized collections, I'll mention that in the fall 2012 issue of RBM, the Journal of Rare Books, Manuscripts and Cultural Heritage, Rachel Hu tackled strategies that we might find useful to measure the impact of our online finding aids and web design, while Anne Boddy and Heather Smedberg investigate ways to assess whether outreach activities in the classroom address learning outcomes. Through that special issue on Special Collections Assessment, the point is repeatedly made that as a community, we need to find common, useful ways to measure the impact of our collections, including those we've digitized for enhanced access. Overall, what I'm taking away from this study is that we need to rebalance our effort. We need to match the energy we are putting into digitizing with the resources we need to sustain, measure impact, and keep those digital resources alive through outreach. Thank you, Lisa. Over to Anne Kenny. Hi, everyone. It's a real pleasure to be here and to have served on the Special Collections Working Group and to have partnered with Ithaca in this effort. I want to start by agreeing with most of what Lisa said, perhaps all of what she said, and particularly the issue around rebalancing. But I think I'd be a little more forceful in stating that. I think there is a disconnect between our stated position on the importance of digitized special collections and the support that we provide to ensure their health. This is not the first time I think we've seen that money doesn't necessarily go where our aspirations are. Schoenfield and Long and the 2010 Library Survey Insights from US Academic Library Directors included a question to university librarians on how they would spend an unexpected 10% budget increase. Only 19% of research libraries indicated they would prioritize investing more money in special collections. Now, you know, we can chalk that up to a shaky economy or to the fact that special collections often come in as gifts rather than as purchase. But the authors concluded might also call into question the assumption that larger academic libraries look to special collections as a key strategy in providing unique value to users. So I think there is a rebalancing and a putting aspirations and future things in line with resources to support them. Lisa notes in the report makes it clear that 90% of ongoing funding for digitized special collections comes from the base budget. If it is indeed inadequate, the way to increase those resources is to do internal reallocation. But the data presented here and elsewhere doesn't really indicate that those tough choices are being made. I understand the desire to find external funds to support efforts such as maintenance, but I agree with the one comment from the survey, quote, growth is limited by our ability to reallocate staff from other parts of the library. As such, I would have expected the report to state the obvious in its recommendations. Libraries must recognize digitized special collections as a vital asset, where the others have much attention as other things. Respondents did indicate support will increase in the future, but it's often easier to put that off to some, that commitment to some future time. Scarlett thinks about things tomorrow at Tara comes to mind. I think the findings do reveal that revenues coming from digitized special collections are relatively modest, hence the need to press for internal reallocation, I think, as a critical next step. And I'm looking forward to the case studies that will be coming next out of this. Second point on diffuse responsibility for digitized special collections. I'm not surprised by that finding, and I wouldn't conclude as the report also considers that it's not necessarily a bad thing. It could in fact signal more mainstreaming, more integration across the whole library system. Personally having one unit or one person responsible may not be the best choice over the long term. I do see at least two focal points or primary responsibilities centers. One focuses on the technical aspects, making sure the access is working, that systems are upgraded, that preservation of the content and the access is maintained, responding to technical glitches as they rise. But the other really is a curatorial one that deals with the content and its context. Digitized special collections need curation as much as their original counterparts. And if we can't provide this, I think there are times when the most responsible thing to do is pull the plug. We are reviewing such issues at Cornell for specialized digital content, created at point based on needs and priorities that really have since languished. I think it's a disservice to maintain and neglected underutilized site of digitized content. Third point on calculating costs, again I'm not surprised it's difficult for institutions to do this. It's difficult for us to do an adequate reckoning for any kinds of costs. We don't do it well in other aspects of library operations. And I appreciated the quote from one respondent who said we don't typically track staff time in ways that allow us to account for estimated costs. And I also believe the tendency is to under-represent or not represent at all the full gamut of those costs. I also don't believe in taking a reductionist view of cost versus value in our library operations and I'll return to that point in just a second. Open access, the future really is toward open access. We're witnessing federal and state mandates. We're seeing responses as a way to cope with spiraling costs and a growing appreciation for the interconnectedness and global nature of research, scientific and creative breakthroughs in this. This openness is permeating special collections as much as the published output. We know data is entering this category so too are primary resources. I'm fully supportive of vendor contracts to recover costs of digitization for time-bound exclusive relations. But beyond that we all benefit from lowering the barriers to access of use. Especially in special collections where the duplication across institutions is low or non-existent and where collections have been scattered. Letters to a person are in one place. Letters from that person in another. I don't want to see a future where we nickel and dime each other to death and for the most part it would be nickels and dimes we're talking about. But I also recognize that key resources are worthy of public and or institutional support to remain open. Final thing, understanding user needs and assessing value and measuring impact. Several things came to mind. Again, we don't do a particularly good job of measuring impact in other areas of the library so it's not surprising that it's not being done well here. How do we assess, for instance, the value of a well-answered reference question? For example, do we tally up all the questions asked and then divide by the number of reference staff plus something added for physical facilities, etc.? No, I think we should tread lightly in assessing ongoing costs based on such an approach. Second, I think we need to consider carefully the notion of community on the web. For universities in particular, there are primary and secondary users. We do consider that audience and their needs from those two dimensions. In the past, much of special collection use was for outside scholars that this was well controlled by having them come to us. Opening our content up to the world can result in an overwhelming amount of resources going to support users who go well beyond our primary constituency and well beyond the scholarly community. When we first put Making of America collection up, the argument was that the cost to serve the entire world was just, quote, noise. But that was only referring to bandwidths. This collection has engendered many user inquiries, technical and non-technical, and most of them have no ties to Cornell, and yet they are assertive in their demands. And as we put those things up, I think we have a responsibility to meeting those demands, particularly as a land grant institution. We're interested in serving the world, but at what price? And finally, considering analysis and impact, what would success look like? Greater numbers of users directing folks to the originals, relieving reference work at the local institution, leading to new breakthroughs in research. Those measures need to be defined both at the institutional level and at the multi-institutional level. Thank you, Anne. Over to Anne Thornton. Thank you. And a particular issue is raised in the report. First, the management organizational issues, and then the motivation for digitization. So first, with regard to management and some of the challenges Anne picking up has just mentioned about diffuse responsibility, sustainability, which really suggests a more permanent state of being, really seems contrary to the experiences of working in this area of digitized special collections and in research libraries generally, where there's lots of flux in the environment, significant organizational change, and new and different roles emerging, particularly for this type of work. And when it comes to the often dispersed responsibilities for managing digital collections, which was the key finding in the report, many of us relate to that and may even have org chart fatigue, I know that over years, from trying to reorganize in various ways to better support these particular activities in the report, which really seemed to come up in nearly every section, I know I certainly was feeling that it's difficult to even tell that, makes me really wonder if anybody feels they have gotten it exactly right. In the strategies for ongoing support section of the report, there is that acknowledgement of departments and the library IT teams are key for integration and ongoing support of digital activities. I suspect that there are a variety of ways in which we are defining what IT work is and what curatorial and librarian work is. And I would also suggest that this has to go way beyond just collaboration and that we probably can't put, we're finding it increasingly difficult to separate library work from IT work. And as Anne pointed out, that's not necessarily a bad thing. In my library, within the last nine months, we have implemented an agile development process. Many of you may have heard of this type of methodology for big products like our digital repository, our metadata management system, and our back-end front-end system for serving up digital content among lots of other things, including lots of smaller projects. And doing it by the book, doing agile development by the book, requires us to have something called product owners in the curatorial areas of the library or in corporate speak in the business units of the organization. And this is, by design, not sustainable within at least our current library organization because product owners, if you do it by the book, must dedicate approximately 30% of their time to represent various stakeholders throughout the organization to collect and prioritize what are known as user stories within an IT development backlog and meet with the IT developers at daily scrum meetings. These are stand-up meetings, so they don't last very long, but they go on for three-week sprints at a time. So rather than a curator meeting with IT staff and developing a long roadmap and typical timeline for digital product development, we have much more of an iterative and focused process for evolving and improving the products. This is a best practice of software development for IT organizations, and it does seem to hold promise for improved products. We've been doing it for about eight or nine months now, but it has really forced us to consider how we staff differently for this in a library setting. It's been a real significant challenge for us to dedicate 30% of individual staff members and we're talking about curators, the curators, librarians, and their time to this, and yet we want the staff who have the most knowledge of current and potential uses of collections and who are closest to the end users to be significant influencers of product outcomes. So how do we sustain that level of engagement and ownership of products over time without just handing off products once they are developed to a certain state? I would really love to know if there are other libraries that are employing Agile methodology with 30% product owners in the curatorial units that are actually finding this method of management sustainable, and if so, how in the world are you doing it? I actually think this would make a great case study and might be something for us to explore since the management and oversight issues are prevalent throughout the report. The next area that I want to address is about the real motivation for digitization activities. As Lisa already referenced, offering greater access was highlighted as that key motivator. And certainly when most of our libraries began digitizing collections, the initial goal was probably about just getting stuff online because of its potential reach. But now it is so much more than that. There are the masses of digitized stuff, but then there are also the more curated digital collection products that engage researchers and scholars, provide us with opportunities to collaborate with users, and even give users an opportunity to enhance our collections and our own knowledge about those collections and create entirely new data sets that may need library attention and sustainability also. How many of us see our digital programs primarily as a way of replacing something from the past? For example, the substantial microfilming that many of us did of special collections decades ago that we could now consider a single collection of stuff by format. And how many of us see our digital library programs as something completely new? For example, digital humanities scholarship with the creation of new tools, which may have grown up and now exist as individual web properties. As libraries are successfully demonstrating that we can accomplish a great deal when our technical resources are focused on this, the challenges of sustaining the work become more and more obvious to us while we continue to innovate on new projects and rethink ways to make existing digital projects more effective and described pulling the plug on products. That's a very brave and really tough decision. But how are libraries deciding when the technical debt that we've incurred from either the originating technological architecture, which was maybe even a decade ago, or the deferred maintenance on our products is just too high relative to the ongoing usefulness of a product, especially if we are no longer investing anything to understand the user needs and the value to scholarship and education of those particular products. It's interesting because as libraries we speak about trusted repositories. A trusted word is key because it relates to sustainability and long-term preservation of digital content. But we have all experienced difficulties in accessing even digitized special collections in digital formats or particular viewers from 10 years ago. The hopeful aspect, I think, and I think the real key motivator is that we also know that the more our digitized special collections are used, the more data they accrue, the more they are linked, the more they are copied, repurposed, remixed, mashed up, and remediated even. And then, their chances of survival are much better within a volatile medium and environment. Thank you, Nancy, and to each of our respondents. We're going to, this is the Q&A portion. This is Judy Rutenberg. I have a question, actually, and it relates to something Ann Kenney mentioned, and I want to pose this to Nancy. Ann mentioned case studies as a follow-on additional project that ARL and Ithaca SNR are working on. In this case, this is a reference to an INLS grant with the two organizations, and I wondered if Nancy could just be just a bit about that project. Nancy? Sure, I'd be happy to. Thanks, Judy. So, right now, Ithaca SNR, and we're doing this actually in-partnership, again with ARL, is very privileged to have the chance to undertake some case studies, which we hope will really let us get inside some of these questions a bit more by identifying a bunch of digitized special collections in organizations of different types. So, right now, and for the next couple of months, we are researching and writing up case studies of eight different projects. One of them are at academic institutions, a couple large and a couple small, and another four are at cultural heritage institutions. And the idea here is to have identified, which we feel we have done, to have identified projects that are doing something extremely successful in the area of sustainability. So, in some cases, it can look like very high usage. In other cases, it looks like the value of their project to the point where they will be sustained by their host institution. In other cases, it might look like some kind of creative revenue strategy. But in all of these cases, they've been able to demonstrate that they've been around for a bit, that they have demonstrated usage and also some kind of stable funding structure. We're researching them right now and finishing them up in the next few months. The final publication will be in the fall of this year. Does that help? Yes, thank you. We're looking forward to that work continuing. I do have a question from our audience and it's for Ann Kenney. Ann at Cornell, what would pulling the plug mean? Would it mean outright removal of digital collections or do you have something else in mind? Thanks for that question. Pulling the plug in this case means looking at particular accumulations of material under we had one special collection that was the pet project in interest of a former president at the university. A fairly short-lived president at the university. It was curated for some time during his being here and through a couple of years thereafter but then it has just languished. It wasn't well integrated into the curriculum. It didn't represent particular interests of key faculty. It wasn't being heavily used and so what we are doing is looking at the discrete parts of it. Some of it was Cornell University Press publications that had been digitized. The approval from the press to do that. Those are actually now going to be pulled into some other kind of environment in which the texts are being made available. Those parts of it will not languish. They will be there. Other parts of it will just go into a nice quiet dark storage. Do any of our respondents want to have any questions or want to get a last word in? This is Ann again. I would like to ask about the set of recommendations and whether I'll feel that this is the issue of reallocation of base budget funding as a critical necessity for digitized special collections. Will it really automatically surface up higher in the recommendations or not? I certainly think that it's surface is great and that if it's surface is higher that's great too. One of the things that I've been thinking about with this is the challenge seems to be about actually counting the investment that's being made and we don't really count staff time that way. I also worry that one of the things that I think that we need to be thinking about is how do we know what investment needs to be made? As Nancy put it, is it just that these are efficient practices and processes or is it that we're not investing or is it that we can't articulate what is or should be? And so I would ask the broader research question of like how do we then determine what the investment should be and I don't have a good idea on how to advance that. This is Ann Thornton. I think that's right. I think Lisa's point is right. How do you determine what the investment should be? At MIPL we have been doing this process called a business request for work with anything IT related. So our digital projects are among them but there are also all kinds of other IT projects that aren't about digitizing special collections that we do this for and in each case we actually do try to determine the cost of the existing staff not just any new staff that might come on for something but existing staff time for the work. But the way that we review these projects is perhaps on a quarterly basis or so and they'll come in fits and starts and not every similar type of project would be evaluated at the same time that other similar things would be evaluated. So it's very difficult to determine that relative to other things and other work that we might take on. I find it really, really challenging. So I think it's a great point. This is Nancy and I would love to hear more about that process offline but I probably should mention also that we've been trying to develop. We developed a version of something called on a previous project and we're still tweaking it with the idea that maybe there could be a set of questions that you would ask of a particular collection first of all about the motivation for creating it and how it's measuring up for whatever that is if it's a certain kind of impact, if it's a certain kind of reuse and then from there what further investment would make this hit the targets we want it to reach to get out to the audience we want it to reach so maybe this is something that we can all share after and I'm happy to direct people to the health check tool that we already have but I wonder if a big piece of the challenge is figuring out which of these do just fine with a very kind of modest level of ongoing work and which are the ones that actually do need something much more robust behind them. Thank you. We have a couple of more questions from the audience I'd like to answer. One is and I suppose this could be answered by any of you who do you think the survey findings and the comments that the discussion here today are applicable and scalable to smaller non-ARL institutions such as consortia or even non-academic digital projects? This is Lisa. I absolutely think so. I was talking with a smaller non-ARL library earlier this year about this very thing that so many of us regardless of our size and regardless of whether we're a library or a cultural institution, historical society, local genealogy resource have put a lot of investment in digitizing but haven't thought too much about the long-term digital asset management functionality so I actually do think that it's applicable to any institution that's digitizing materials and providing access on the Web and I think we're also seeing kind of a maturing of some of these projects and some of the projects are starting to look very dated because none of us have gone back and spruced them up pulled them into new discovery environments and I do think it's a real issue across the board. I was just going to mention that one of the things that because this literally was the population was 100% ARL institutions, I think the questions absolutely we've seen them apply to cultural institutions, not just academic and smaller, not just larger. My feeling is that the findings we don't know what they would be in other sectors. I wouldn't at all be surprised to find the same set of challenges surfacing. I feel like if the largest ARLs are still struggling to identify sources of funding I have to imagine it's just not any easier if you go as you go down in budget. So my guess is that some of those they may be doing less activity and maybe having even more trouble supporting the activity they can do. On the other hand it is expensive to digitize and the ongoing support is not insignificant. Smaller institutions may be much better than larger institutions in sort of sharing the collective asset and responsibility. We have seen on a pretty grand scale for research libraries the development of HathiTrust Center for Research Libraries to help support some of that work. But I think that smaller the five colleges come to mind they have banded together maybe more effectively than we have in this sort of last frontier of individual distinctiveness. I'm so glad you mentioned that point which gives me a tiny opening to mention that in fact a couple of the case studies are actually taking this on. One of them is the Triptych platform so we're looking at a collection at Haverford College but specifically with an interest to understand how some of these shared platforms can work. And another one on the cultural side is the main historical society which has developed a platform to kind of bring in even the tiniest historical societies in the state of Maine. So I think you're right there's going to be a lot of focus on what kind of community solutions might be. I think we have time for one last question and we have one additional one from and I'm concerned that only 25% of the users and this has come up several times are identified as a core audience. Is this a good thing? Does it mean that digitized special collections in fact have a long reach? And is this a difficult conversation with university administrators? I think if I were, this is Ann Kenny, if I were to do more assessment of user needs and expectations and how best to increase their the value of such materials to their experience, academic experience especially as we look at at the undergraduate level greater and greater emphasis on research at the undergraduate level as well as at the graduate level. So that's where I'd focus more of my attention. And this is Ann Thornton even at a library that isn't within a university setting. We when we begin a digital project we, who is this for? Particularly if there are special features or that we want to focus on developing to improve data around the collection and we're you know, and still want to know who it's for so that we can I mean, I think we've had a tendency in the past just to proclaim upon launching a new digitized special collection or product success once it launches. And I think we're beyond that now in a healthy way where we're really trying to be very clear at the outset and I can change, but clear at the outset who is this for, what are we trying to accomplish here? And it's often and thankfully so more than just eyeballs on the website. Thank you all. We are just about at the end of our hour so this concludes today's webcast. I'd like to thank all of our speakers for an excellent program and everyone in the audience for your interaction and for tuning in today.