 I am Tim Cole from the University of Illinois at Urbana-Champaign and my job this morning is to briefly introduce and provide context for the findings that Katrina and Harriet will present. They'll be briefing you on the user research outcomes from two projects involving digitized special collections hosted at the University of Illinois Library. This is early research, a first step in investigating among other questions, the question posed in our session title, but it's just a first step. We will not offer a definitive answer today to the question we pose, more research we feel is needed, but our findings should be taken in context and alongside similar research from elsewhere. First things first, before diving into what we did, let me begin by acknowledging the generous support of those who sponsored our research, especially the Andrew W. Mellon Foundation, the National Endowment for the Unities, and the DFG, the German National Research Foundation. My way of context, let me begin by summarizing our assumptions and some of the limitations of our work. Given the early state of our research, the community's research, into these questions of the benefits of linked data for end users, we took our initial stab, a small-scale case study approach, looking specifically at the initial reactions of domain scholars and graduate students to LOD-based enhancements of three established web-accessible digitized special collections hosted at Illinois. While this limited our ability to generalize our findings, the approach gives us access to advanced users, involved content, especially rich in links, provided a scope in keeping with our resources, and yielded data points that we feel can be used in combination with results from similar studies, both larger and smaller, to begin getting a clearer understanding of the potential near-term benefits for end users of LOD. More about these three collections that we used in just a moment. Those of you who may remember seeing the Gartner New Technology Hype Cycle, shown here from about 15 years ago, are aware that it's not really a cycle, nor is it all that represented in detail of how technologies in general are received. Rather, it's an exaggeration, a caricature, if you will, of what happens when the next new technology comes along. But like most caricatures, there is an underlying grant of truth. It's very easy by turns to both overestimate and then underestimate the potential impact of linked open data in libraries. To avoid this, the key we felt in grounding our assessment and initial evaluation was to ground our initial evaluation in practical experience. As we begin the introducing LOD-based features and functions and services into certain of our digitized special collections, we chose to focus on assessing the potential benefits of connectedness. There are many aspects of linked open data. These are the ones that we thought to focus on. For example, how could we use linked open data models and services to link to context about our resources, and importantly, to the person, place, and event entities associated with our resources in order to get more information about those entities. We also wanted to explore the ways LOD services might facilitate the use of distributed authorities and analysis of simple patterns of connection surfaced by media from these collections. Our mental model of user experience also needs to be explained to provide important context for understanding our findings. User research, as we are discussing today, is about more than just interface design. It's about more than just helping the naive user navigate a library online catalog. Not that it's unimportant, but in this case we wanted to look at more advanced uses and users, especially when it comes to digitized special collections, which are focused on that kind of audience. We must understand the information, seeking experiences and needs of domain scholars and students. We must understand how linked open data can change the value proposition of these resources. To do this, we need to consider the totality of the user experience as illustrated in the Peter Marvel's user experience, Honeycomb, shown here. You can build a clean, easy to understand interface, but the resources and functionalities behind an interface are not useful, valuable, or of interest to the researcher. It won't matter for that audience. So our goal is to understand how to make our collections more useful to scholars, helps scholars engage with our collections, helps scholars connect our collections to other content of interest to them. Intuitively, LOD seems well suited to do this, but is it? Do we know how to use LOD effectively to do this? So how do we go about investigating this? Do we know how to research? What kind of methods do we use? As those of you who don't use the research know, it's typically not enough to simply ask scholars what LOD-based features and functions they want in the abstract. You need to give them concrete. You need to be concrete, give them something to react to, something to build from. Harry and Katrina will go into more detail, but we rely in general in these studies, primarily on semi-structured interviews and wire frames, and usability testing, one-on-one usability testing to gather feedback from scholars. Briefly, a couple of words about the collections, and again, Katrina and Harry will flesh it out a little more. The Molley Collection, which is the theater throwing arts collection, includes more than 4,000 digitized set in costume designs created by the Molley Group that was active in New York and England during the middle of the last century. We identified person, theater, and play that is mentioned in the descriptions of these set designs and costumes. And then we tied these to other services such as VIAF, Wikipedia, and other web-related resources. Activating these links in real time as we build search result displays allows us to generate sidebars and knowledge cards for context and include links that users can use to get even more context about the actors, directors, plays, and so on. The Colbe Proust Archive is a very different kind of collection. It's comprised of the research notes of the late Philip Colbe, the faculty member of Illinois, who edited the 20-some volumes of Proust's correspondence, and spent about 40 years researching the life and times of Marcel Proust. This data was particularly rich in person and bibliographicalities. Many of the people were not authors, so VIAF was a little bit less useful, but there were other sources in the web that were very useful. And these personalities co-locate individuals from the most prominent families of Paris during Proust's lifetime. So we experimented as an initial stab at this with civil visualizations as these social networks in order to gauge whether our researchers thought such visualizations might be of interest or might suggest ideas for more sophisticated analyses and visualizations. And then finally, Harry will talk about Elmeticon Ly, which is essentially a virtual library of digitized album books from six major rarebook collections in the U.S. and Europe. Elmeticon provides integrated search, discovery, and access to 1400 digitized books and includes descriptions of more than 33,000 individual emblems from within those books. Here again, we mind VIAF in real time to provide information about personalities the emblem authors, for example, associate with digitized books. But also because emblem studies community agreed more than 20 years ago on subject indexing using the icon class Tharis, we were able to use icon class LLD services to offer icon class search engine browsing and any of the four languages supported by that service. To be clear, we do not maintain a copy of icon class locally. All searching and browsing of icon class is done against copy of icon class maintained in the Netherlands and then applied against our database, which simply has the numeric codes for the emblems that we have in our collection, that we index our collection. So that concludes my introduction. I'll now turn it over to Katrina, who will describe research findings and possible next steps based on her work. And let's find the right presentation. And you want to find the one for Katrina? It should be opened down the bottom there somewhere. Thank you. Good morning, everybody. I'm Katrina Fenland. I'm an assistant professor at the University of Maryland iSchool. But I'm a recent transplant from the University of Illinois where I worked with Tim and Harriet on these kinds of projects. So I'm going to dive into an Andrew W. Mellon funded project. That was about adding linked data to a couple of special collections and then evaluating potential benefits for users of the features that are based on linked data in those collections. So I'll start. I'll give you an overview of how we did the study. And then I'll talk about the three features we added. And then I'll go into our outcomes. So both their reactions to specific features and also some of the more general takeaways about potential benefits of linked data. Tim went over our test collections, the Motley Collection of Theater and Costume Design, and the Cole-Pruist Archive. We did this study in two phases. So the first phase was about conducting a series of interviews to set a benchmark for a second phase evaluation. So the first phase done before we had added any linked data to the collections, just a benchmark evaluation of our two test collections. Then we spent some time adding linked data, features based on linked data to the collections. And in phase two, conducted more interviews to get a sense of their reactions to the enhancements that we'd made. So 10 interviews in the first phase with humanities researchers, most of them grad students, in an array of fields, theater, history, art history, medieval studies, literature, languages, and then implementing linked data and features based on it. And in phase two, 10 more interviews with researchers to get a sense of their reactions to the features. But we also made an emphasis in the second phase on trying to draw out their practices, their research processes, especially those that might benefit from linked data. So asking them about how they make connections among primary sources and external sources, secondary sources, how they navigate among distributed collections. So really trying to draw out their processes in addition to their reactions to the features that we had added. So I'll talk about what we found out. I'm going to start by walking through the three features we added and the scholars' reactions to those features. And then I'll come back to more abstract claims about potential benefits of linked data for collections. But the overarching theme here that's kind of casting its shadow on everything that I'm going to talk about as an outcome is that our participants were really enthusiastic about the features based on linked data. But their enthusiasm was qualified in really different ways by contextual factors that really need more investigation. So the three features that we added. We enhanced item level metadata records with contextual information about entities that we found in the records. We enhanced search results with a knowledge panel that we generated based on linked data in response to users' queries. And we added an interactive visualization of relationships in the Cold Pruised Archive. So I'll walk through each of these. Here we have the item record enhanced with information about contextual entities found in the item record. And so that context is added in two places, both in links within the record itself and also in the sort of sidebar that takes snippets of information drawn from external resources through linked data about entities that are found in the record. Here's a zoomed in view of one of those snippets. So drawing on external resources like Wikipedia and authority files and IMDB to sort of flesh out the record with contextual information. What did scholars think about this feature? So in terms of cons, they found the information we added, the links that we added, the contextual information we added to be underspecialized for their research needs. So they want less general, more particular kinds of connection and information. But they want it specific to their needs as researchers and where they are in their research process. However, they liked this kind of thin layer of context, especially if they were early in a research process still kind of doing the exploratory work of figuring out what's even out there that can support their inquiry. So they liked this for early phase research. They also like it for getting information about entities or concepts that they consider peripheral to their immediate information need. Of course, that's a really subjective thing. So figuring out which things should be contextualized is kind of a challenge. Okay, second feature we added. So this was added to the presentation of search results in response to queries. This is a dynamically generated knowledge panel. Might be familiar to you from your Google searching. So they've done a search for a term, Thomas Ducker. They get the results related to Thomas Ducker at a list as usual. But they also see this knowledge panel that offers more contextual information about entities related to Thomas Ducker that have been derived from or inferred from linked data. Okay, what did they think about this? So in terms of contextualizing terms they were searching, they found this kind of knowledge panel a little less useful because most of the time when scholars are doing a search for an entity, for primary sources about a given entity, they know a fair amount about the entity already. So the sort of, again, the sort of shallow contextual information about this entity and related entities was less useful to them. But what they liked it for was for instant disambiguation of their search terms so they knew that what they were finding was what they were searching for. The person they were finding was the person whose name they had searched. They also really liked it. This was interesting to me. They liked this. They expected this to give them an overview not of sort of general state of knowledge about Thomas Ducker on the web and people connected to Thomas Ducker. They wanted that to be an indicator of what our collection held about Thomas Ducker. So the knowledge panel as a way of orienting them to the scope and contents of the collection relative to their specific information need. This is interesting because we know from decades of user research that historians and other scholars always start their search process by doing exploratory searches to feel out what's in a collection. And this offers an effective way to kind of tailor the presentation of the scope of the collection to them based on link data. Okay, third feature that we added. This one a little more analytic and interpretive. So this is an interactive visualization of relationships among entities in the Cold Proust archive. So interactive in the sense that scholars can adjust this timeline to see the relationships at different periods of time. They can pull out individual relationships, click on notes to learn about the families and the people that are connected to each other. They can also annotate this graph so they can add notes, two nodes into relationships about their interpretation of what's happening in this graph. So what did scholars think of this? This was the feature they were most enthusiastic about. They thought it was useful for a number of things. First as an entry point into the collection. So an alternative to searching directly for primary sources. They wanted the nodes to link back to the primary sources that they had been derived from. So they could use this as the entry point. They were also interested in using it as a hub for collaborating with each other. So having sort of a limited audience for this work but their peers so they could communicate through the annotation process with one another. And we found that they liked this because it offered new opportunities for analysis. This kind of graph allows them to pursue lines of inquiry that otherwise would be impossible given the current representation of the collection. Now this is interesting to me because we've often thought so traditional conception of collections is as sort of raw information for them. The responsibility, the duty of interpretation analysis is squarely on the shoulders of the user. But here we found that they were very happily sort of asking us to do some level of analysis that they could get a better start on their works. They wanted us to do some interpretation for them. So overall we find our participants were hungry for new modes of access to collections. Even collections that they were very familiar with are new well. They wanted more connections across collections to be accomplished through, that's what they expected the links in the link data to lead to more related collections elsewhere on the web. More contextual information for primary sources, for search results, also for collections as a whole, and more links and analytic features. So the takeaways from this study, more contextual information is better about different levels here. So primary sources, entities within primary sources, the collection as a whole, especially if that contextual information is linked and actionable for the scholars. But it's qualified. There's a great deal of variation among exactly what kinds of context people wanna see. And we've known for years that there's variation among disciplines. But this study really highlighted for us how much variation there is based on more subtle things. The phase of research that a scholar is at, so whether they are just starting out or whether they are deep into a book project, the immediate context of the current search, so what they're actually doing on a given day matters too for what kinds of contextual information they wanna see presented to them along with the primary sources. So in an ideal world, the presentation of contextual information would be really responsive to where they are, who they are and their prior knowledge and their progression through their own research. So what they see surrounding the sources they're searching for on one day might differ as they progress through their own understanding of what's happening in a collection. This study confronted some really familiar limitations for user studies in these kinds of cultural collections. The scale necessarily small and local because of the methods we used and the scale for the project. It's always hard to identify a research community that's using a collection so that they don't have to sort of imaginatively put themselves in other shoes as they're responding to the collection, right? So trying to identify a community when the collection itself is a niche collection that really caters to a very specialized, highly distributed international audience. So hard to get reactions from those users. And of course these studies tend to be when you're looking at sort of developing new features the studies tend to be speculative. So you're asking them to respond to potential features in a collection and how they might use them in their research processes. Our future work is really about trying to fight back against some of these limitations. So there's the basic work of identifying and implementing really meaningful connections among collections and external resources through entities in the metadata. The things that scholars care about in the metadata. But as far as sort of getting back at these user studies and limitations we want to find ways to identify the most useful kinds of context for scholars at different phases of work, doing different kinds of work, trying to answer different kinds of questions not just based on discipline. And that kind of rich user study is gonna depend on longitudinal study, lots of iteration, the triangulation of results from multiple methods. So both perspective methods forward looking, asking them to speculate, but also retrospective ones really looking at what they have actually done with primary sources in their publication or in their own reflection. So someday with this blue sky, ambitious goal of being able to present collections to them, aligning the context we provide with and the connections that we provide through linked data with their specific use context in real time. So that's our goal. Okay, and I'll pass it over to Harriet to talk about emblematic, okay. Thank you. Sorry. Great, thank you. So today I'm gonna give a brief talk on an older project called emblematic online. And so this is more of a kind of a general research study and talking a bit more about the discovery, things we learned about discovery for that project. So first set of context though as we've heard from Katrina and Tim that digital archives today provide unprecedented access to these were archival materials but also enhanced exploration as we've seen with data mining, image analysis and other approaches for data analytics. But again, when I embarked on this study in 2014 there were still large gaps in knowledge on how the digital resources were used and research and teaching and the efficacy for the digital collections for scholars. So today I'll briefly describe, first really describe the background rationale at the time of my work in investigating user requirements for digital collections and then how I investigate key issues of usability via the case study or emblematic online. And then talk a little bit about how my analysis of how humanity scholars specifically how they used emblematic online and research and teaching I gained insights on a diverse array of user needs for digital collection. And today I'll highlight what we learned about user's needs in particular for discovering access and using the enriched metadata and icon class and the role of discovery functionalities in their work practices for digital collections. So again, it's brief background context. There's a rich corpus of research on the use of digital collections by scholars. So the papers that Tim and Katrina linked to have a number of the studies cited on humanity cyber infrastructure studies of disciplinary research practices that Ithaca has done that highlight how scholars use digital contact and complex and multifaceted ways. But in the creation and curation of digital collections we're learning that it's not enough to simply have that static scan content and expect users to flock to it. And so this Ithaca, this older Ithaca report a test user engagement and attestment are key to making our digital resources maximally useful. So with the strategic assessment we can build digital collections to promote use of the content by students, faculty and public. We can also improve our existing collections in ways that facilitate the integration of our digital resources into their scholarly practices and also ensure a reasonable return on investment of all the grant and institutional funds that we're investing in our collections. So thus we ultimately want to know how can we connect the structure and content of our digital collections to actual scholarly practices? And that was the question that motivated my study of emblematic online, which provided a rich case study of how users engage with digital archives from a variety of disciplines. So emblematic online was built as a multi-institutional international project that received multiple NEH grant awards and aims provide comprehensive access to digitized copies of emblem books from leading rare book collections around the world. And so very over the years many emblem books have been digitized to varying extents and emblematic online brought these multiple collections from the institutions listed on the right and what we call the open emblem book portal. So it's had a couple of phases of digitization. The first one was digitization with the NEH DFJ grant and then the most recent development of the emblematic online project and expanded to include the multiple digital collections and also looked to enhance the metadata further with more extensive application of icon classes as Tim mentioned and improving the search interface. And just really quickly, here's what we're talking about when this is what the scholars that I work with were looking at and this is what they use in their scholarship. So this is an emblem and it's a genre of early modern printed works emerging between the 16th and 18th century and it comes primarily in this tripartite form with the motto which is pretty self-explanatory, the pictora in the middle and the subscriptio which is a longer tax or company annotation. And the pictora as you can see is a heavily symbolic image with multiple elements. And when you just oppose the symbolic picture with the motto and with the subscriptio, you get a fairly cryptic and complex image and this is what scholars are studying. And scholars from across disciplines use these emblem books for rich integration of visual culture and textual culture and what they reveal about intellectual thought in the period. So these emblems are in emblem books and these emblem books are in special collections around the world. And for a long time the emblems were primarily found through seminal but incomplete print bibliographies. And what emblematica sought to do was, as Tim described, adding metadata and making these searchables so that scholars can find these richly textual and visual digital content. And so emblematica was also, I would like to highlight an in-depth collaboration between Mara Wade, Professor of Demandic Studies, Tim, MJ Han, Tom Kilton and then later me joining on the last phase of the project. And early modern scholars such as Peter Daley argue for the importance of librarian scholar collaborations for digitizing special collections such as this one. And the key reason is to enable this user input on top of, to begin with and then augmenting that with the usability study that we embarked on. So by examining the needs and practices of emblematica online users, I started to examine how emblematica online's functionalities match the scholarly practices of humanity scholars as they use these digital facsimiles of these rare book materials. And so just briefly, these are the goals where to understand the research practices of humanity scholars working in early modern studies in particular, understand their behaviors and then also gather input to assess the new functionalities and services that were added to emblematica online in the newest iteration of the portal and determine future functionalities to further enhance the portal. And so the methodologies was two phases. I conducted 10 semi-structured interviews with researchers, students, graduate students in teaching faculty from Illinois, Wolfland Biddle and at the Emblem Studies meeting in Germany. And then they were from the disciplines of literature, linguistics, music and art and their experiences range from minimal use of emblem books to extensive use. And then I did five observational usability testing sessions with faculty and students from the University of Illinois and the testing protocol was based on the interviews and the participants conducted a series of tasks across the whole emblematica online website and open emblem portal. And the analysis of our scholars interviews revealed key insights into how humanity scholars integrated the collections to the research workflows and how the collection could be iteratively designed to meet their needs. So I'll start with talking about the interview findings and then wrap up with what usability testing highlighted. So of the interviews in particular, collection enhancement was one of the most prominent benefits cited by respondents. Because the digital collections from the single institutions were now magnified into this aggregation, respondents noted that granular search functionalities in particular with icon class could enable scholars from multiple disciplines and other research areas to discover the works for their own research and expand the use of emblem books as a scholarly source. And with the breadth across disciplinary research that could draw upon this emblem collection, previously under read scholarship and emblem studies could be brought further in dialogue with scholars. So that's collections with these robust discovery mechanisms in particular, facilitated, they saw both discovery and scholarship with these collections. Along these lines, a key area that respondents identified was with emblematic online was a critical role in discovery. So emblems are notably unique type of research because they're heavily dependent on the visual content as much as the textual content in the multiple languages, as Tim mentioned with icon class. And so having the search functionality enabled new methods for searching this archival content, enabled the scholars to look at specific emblems and again expanding interdisciplinary research via new sources of content such as the emblems. So as the first quote from the interview mentions, scholars who see that the digital collection could encourage new approaches to search and the searchable functionality with the linked data could enable scholars to surmount linguistic barriers as well as using the visual elements of the content as search terms. And the metadata is also key to supporting the research behaviors with this type of unique digital content as it allowed scholars to encounter the emblem books that they previously may not have known of as a primary source for their scholarship. And then thirdly, we saw that the linked data and the searchability of the content also went to humanities data curation. So as the scholar noted, data management for the humanities could possibly be even more possible with the power of rich metadata as the scholars could build corpora of digital content and what they could consider as data sets and allow them to share with other scholars and prove their hypotheses behind their own close readings. So now briefly discuss the usability testing. So we did five user testing on the observational user testing sessions with humanities scholars on the functionality of emblematic online. And this is the old interface of emblematic online that we use for the user testing. And so just a couple of things that I thought particularly highlighted the usability in terms of discovery and using the metadata. So one of the most frequent challenges that the users encountered with the old interface was how to start searching for emblems from the homepage of the website. So from the opening page, some participants could identify the search bar but it was not as intuitive in terms of how to begin their search. So the overall structure of emblematic online was a mixed experience for users in terms of the navigation of the tabs and the ways that we originally used the metadata to filter their search results. So this revealed that he concerned for emblematic as we transitioned into a broader tool is how do we make the digital collections as searchable and as readable as possible so that the researchers can navigate intuitively to the digital content they wanted. Another issue was discovery of the content and the mechanisms provided. So participants immediately began with simple keyword searches to find emblems that match authors, topic and iconography. But participants noted that even though the search interface was customized to search specifically for emblems, they still might use features, use in other databases such as early English books online, including a need for a legend to explain the portal's search limiters for transcriptions and also the icon class terms as a way of searching that they weren't quite familiar with. And thinking of icon class, how specifically did they react to the use of icon class as a discovery tool? So icon class as convention is a multilingual classification system specifically used for art and iconography. And so over 23,000 of the emblems have been assigned and applied icon class heading terms. And most noted that icon class had immense added value for browsing the content as the quote, a test from one of the usability testing participants in terms of being able to drill down and find the topics they need. But we also learned there's a tension in implementing a new system such as icon class as respondents requested more of a context on how to use it. So we learned that when building digital projects, the complexity of the tool really must be balanced with kind of the navigability and helping people learn how to use metadata such as icon class. And as the scholar quote also hints, humanist's desire to search content in multiple ways, not only in direct of searching, but in serendipitous browsing, which icon class started to lend them being able to do by able to find emblems that were linked with the different terms. So this is the new interface that resulted from our finding along with work by a UX design firm. And there's a further emphasis on searchability functionality as key to this revamp. Because we founded our testing that the expectations of users when exploring digital collections is complex, ranging from a basic need of high quality reproductions to being able to distinguish the types of archival content they're perusing. And both the testers and the interviewees frequently express the need for context, annotated context and unannotated content and other functionalities that allow them to fully engage with M1 books as archival scores and scholarly area. And we also consider the needs of interdisciplinary scholarship as they take advantage of easy access to these vast digital collections of content and how to address the needs for this broad scholarly knowledge base that allows all types of scholars to interact with this digital collection. So these are the summary of what we found as user needs. These are also highlighted in the portal article that we published a couple of years ago. So the core user needs, especially in terms of discoverability, is enhanced discoverability and accessibility, interoperability to facilitate interaction across multiple digital collections. And then also interactive center digital curation so that users seek to interact with the digital collections to do comparative analyses, export content and work with content across multiple collections. And this speaks for needed, for greater capability interactivity with collections. So just to really quickly wrap up, these are just some of the future impacts of digital collections and what link open data may have on scholarship. And in particular, I wanted to highlight how could digital collections with rich metadata and linked open data lead to an expansion in scholarly communication as scholars engage with other scholarship and scholarly outputs through exposure of primary sources like M1 books. And just to reiterate what Katrina was saying, further user research is needed to reveal the fuller range of ways that different levels of users are able to interact with our collections. Okay. So we have about five minutes, roughly seven minutes for questions. Hopefully you guys can relate to your experiences and what you found useful. There has been a fair amount of publish on this work. Magwork is the first three items there. Those links to be in the slides which we've posted, include conference papers, a book chapter, article, a project white paper, that Katrina and I co-authored, as well as the poster I presented at ASST a few weeks ago was available. These websites, not all the features we showed are available in the websites that are up right now. The American one is probably the one that has the most features incorporated into production. The image search test one is just that, a test way to get into the modeling collection. Code proofs is up, but the visualization in particular is not available. And if you have questions about anything today, our email addresses are there. So, comments, questions? We are two minutes, three minutes over time. So I want to thank everybody, and I hope this was useful. Please join me in thanking the panel. And we're delighted to spend a couple minutes here.