 Today, this is Dr. Bill Fisher. I'm coordinating the School of Library and Information Science Colloquium Program this year, and this is our second program for the fall term. And I'm delighted to introduce John Dove, who is with us today. John's a senior publisher with Credo Reference. And he'll be telling us a little bit about what Credo is and what they do. John has an extensive background in electronic publishing and online education, and has been doing this for a number of years. He was part of a startup in 1968 that made accessible one of the first online databases. In this particular instance, it was stock market information. And he has been in the sort of online educational business ever since. He's been with Credo for about 10 years now. And we're going to hear John give us some information about what search engine designers could learn from their reference interviews. So John, it's all yours. Great. Thank you, Bill. And I'm glad to be back speaking at Canada State. The last time I was there was maybe five or six years ago, and it was less Boston. And by the time I got to San Jose, there was a huge snowstorm in Boston. So I was so happy to be in San Jose. But now this is a beautiful fall day here. So let me just say that, you know, in some ways, I was destined to be in this business. My mother was a life, in some ways, I was destined to be a researcher for Encyclopedia Botanica. And so it's clearly online encyclopedias was something that was destined to be in. It's too bad she didn't get to see that day because when I joined that start-up on Wall Street in 68, her reaction was, oh, John, what are we going to tell the family? I thought you'd do something useful with your mind. So I think she'd feel that I had come around now that I've devoted my professional life to online encyclopedias and the supportive libraries. This talk that we're giving is part of an ongoing inquiry that we as CREDO run as part of our understanding of the market. And let me just say a few words about CREDO as a company. The words I'm going to describe are perhaps not entirely unique in terms of business, but they are unusual. Our lead investor, and he was also the founder of Silver Flatter, where I worked a number of years ago, talks about building a company that is in ecology. And like any good ecology, it's vibrant and growing. All the constituent parts of that ecology are experienced themselves as well-served by participants in that particular life. And so the role of management in such a company is not just to sort of decide what people should do and tell them to do it. It's much more to engage in an ongoing inquiry with employees about what it's like to work at the company with customers and our case librarians and their users in terms of what would really delight the users and really serve the needs of libraries and with publishers who are a very important part of contributing the capabilities that we bring to the table. And so this inquiry that we're going through today is part of understanding how to users find the world that they face today. And I was very glad to be able to get some of your input in terms of the survey that we did, and I encourage those of you who are watching this later to indeed go to that survey monkey and continue to put in information. It's very useful to us. So we're going to take a look at Google's vision for what users need and compare that with, you know, libraries and library systems and then also look at how Credo approaches the same thing. And this is the inquiry, as I said, that we've been going over the last number of years and it's obviously a number of moving targets, moving targets in terms of what users need and moving targets in terms of what Google provides, as well as what Credo and other providers of library systems do. So I've added in a couple of things at the end of this presentation, which I may, we may not have time to go over in detail, but I'm leaving in the presentation for those of you who see this, you know, see this offline. And one is an op-ed piece that I just recently wrote in Against the Grain about filters. And secondly, I know that a number of you may be interested in Credo's internship program. San Jose State is one of the main providers for us of internships and we work with Sandy Hirsch, your dean, and others to come up with really good projects that are meaningful to us in our business as well as meaningful experience for you. So there's a number of slides at the end that we'll go over if we have time, but I'm leaving in the presentation just in case. So let's start off by contrast, comparing and contrasting the world of Google and the world of librarianship. The man on your left is Terry Winograd. And by full disclosure, I need to mention to you the fact that Terry has served and has for a number of years, served on Credo's corporate advisory board. He's been very, he's very well known in terms of user-centered design at Stanford. Terry was interviewed by National Public Radio and basically asked, so what does Google really want to be able to do? And what Terry said was, what you really want is the mind-reading machine. And the mind-reading machine will know what you want. It will know what things you've searched for recently. So if you've searched for, you're searching for China, but just a few minutes ago, you were searching for Japan, then it probably means you're interested in China as a country. But if your previous search was for dinnerware, then it's probably that you're, you know, knives and forks, then you're probably searching for China for dinnerware. So if you take into account your search history, then the search algorithms can do much better at predicting what it is that you're really interested in doing. So he described this mind-reading machine and that was, and he indicated that he's a consultant who works with Google. In fact, his two students, Sergey Brin and Larry Page, then he left his PhD program to go start Google. So then I'm not thinking we're going to get the second audio either. The second audio was a response that came in from a listener. And the listener said, well, it's interesting to me to hear that your scientists are trying to figure out how they could build the equivalent of some big electronic brain that would somehow know what it is that you want and could answer all searching questions. So I hope you're sitting down because I actually can solve that problem. The series on search engines brought a suggestion from Amy Hartman of Toledo, Ohio. She writes, I was struck by frustrated scientists and business people saying that if they could just develop the equivalent of an electronic brain which could interact with people to help them find what they really want, that would solve all searching problems. Well, I hope you're sitting down because I can solve that problem. I am a public librarian. I do not require you to look at advertising. I'll keep asking you clarifying questions until I feel I have a complete understanding of what you're looking for. Why not give us a try? So we dig this contrast exactly the situation between what could be done by a Google or a credo or anybody else who is a computer system and what can be done by the human interaction that in the reference world is known as the reference interview. If some of you may not have studied yet the reference interview, but you will if you're in a library program. And actually really important is there's a seminal article in the reference interview literature is one from Samuel Green who was a public librarian at the free public library in Worcester, Massachusetts. And he wrote in the first year of American Library Journal back in 1876 an article about you should pay attention to users when they, readers, when they come into your library and it's a fabulous article. I highly recommend it and he basically ends up describing 20 different people walking into the library and for some he takes them right to the resource they want. For others, he recognizes that this is a teaching moment, that this is an actual opportunity to give them some literacy about the use of the library itself. So rather than answering their direct questions, he leads them to the catalog and shows them how they can answer their questions with themselves. And it's a wonderful article. But indeed, I think the key question is, you know, why is it that, you know, how would you compare what what what computers can do and what humans can do? By the way, if any of you are looking for a mind reading machine, there was one on sale from eBay several years ago, it was it was produced in 2265 and teleported to the common to this common era time. And so there may be another one being produced later and I'm sure you'll snatch it up and then then you'll have your own reading machine. So rather than thinking that Terry's comments that I paraphrase are just Terry's here is a statement from Peter Norvig, who's Google's CTO, director of research, and he basically says, gee, you know, many years from now, you'll just you'll interact with a digital intermediary, offer a few suggestions and lo and behold, we'll come back an entire report just the way you wanted it, with exactly the level of detail you wanted. Now, you'll notice that there's very little in his description that involves a librarian. And that's one of the key differences that we find between Google's vision and Credo's vision is that not that we want to replace a librarian, but we believe that actually the librarian brings to the table a significant understanding that is that is left out with just what the information is that Google has. So if we look at that, actually, there's a lot of research that was done trying to compare query systems with reference libraries. You know, you can look at this literature. A lot of it is definitely apples and oranges. It's they did things like watch people coming back from a reference interview and if they were smiling, then it was considered a positive result. If they resubmitted their query to a query system, that was considered to be a negative result. So I don't think you can really put these numbers on this. I think this is a bit of a fool's errand to go after understanding the difference between the human interaction of the computer and human with a human. But I think we can definitely say that the librarians are better at this. There is certainly the key question of why users under specify their queries. It's fairly famous in the literature around the reference interview that someone will come up to the reference desk and say, what do you have about fish? And the librarian is stuck with what does he mean fish? Does he mean fish that swim in the ocean? Does he mean actually he's mistaken and actually he thinks that Moby Dick was a fish, just as Melville did. Or maybe he's talking about Joe Fish is running for Congress in the local election. So a lot of, you know, even frustration is characterized in terms of how some people are so inarticulate in bringing out their question that they won't put what they even know on the table. And there's a lot of reasons why they might not. They may just want to see how much their information there is. So they don't want to over specify their request because they want to have a full choice of what they want to see so that they can direct the drill down. They may not have the effective vocabulary really. And we're later going to see an example from Credo of drilling down into some information where clearly the inquirer may not have known the vocabulary, but learns the vocabulary in the pursuit of their inquiry. So also sometimes it's an emotional thing when the person is asking a question to a librarian, there's very often a power relationship. They've come up these granite stairs to go into this library and maybe they're maybe they're 10 years old and so it's an age difference. And so a lot of what's done in training on the reference interview is to make sure that the librarian gives back this sense of confidence to the inquirer. You've come to exactly the right place. You've asked a really good question. Now let me just clarify a little bit of what you really mean. So this emotional aspect of making sure that the patron or the user, the student feels empowered by the relationship. So it's not clear how query systems can do this, but it's clear that that is part of the experience of the reference interview. So a key question in this is, gee, how well would a well designed online reference service take some of these questions into account? It's also fair to say that an important development within Google has been that the autofill feature and the autofill feature has done a lot in the area of some general Google searches to help with this problem of under specification because suddenly you're given a set of choices and how those choices are drawn upon is related to how much Google knows about the population of users and their questions. I'm curious and even you want to enter in something into the chat about how well you think autofill works and is there a distinction between how well it works in everyday world searching and academic searching. So I believe all of you can see the chats that are presented. So I'll encourage you to enter that information now as to how you found autofill in Google and whether that in fact has answered a lot of this question of under-specifying queries and does it work equally well in general searches as it does in academic kinds of searches. So now I want to back up while you're filling in some of those answers. I'm going to back up and look at this question about, first of all, I'm going to say just categorically reference librarians are better at this than computers and always will be, at least in my lifetime and your lifetime. So and one of the reasons is that there's so much context that the librarian knows that even before the questioner has opened their mouth. So first of all, this reference interview or an inquiry on a computer system is a nexus between two unknowns. The inquirer has a general idea of what they want to know, but they don't really know what there is to know. And the person or system or whatever else that's answering them knows a lot about what it has to know, but doesn't really know what the user is interested in. So there's this nexus between these two unknowns. In the sixties, Robert Taylor, who's done a lot of work around the reference interview, came up with five elements of context that the reference librarian is going to know even many of these, even before a student has opened their mouth or the patron has opened their mouth. So I'm just going to show you a little thought experiment that makes it really clear about how this works because I'm going to show you a reference desk and picture in your mind, I'm going to talk about two examples, the Parsons School of Design. It's probably one of the elite design schools in the country in New York, it's in Manhattan and students coming to that reference desk at the Parsons School of Design have, you know, if it's a beginning of the semester, they've got 14 months to finish an MFA at the top design school in the country. On the other hand, at the West Virginia Northern Community College, it serves a lot of people in that area and indeed, a lot of them are getting allied health kinds of certifications and taking a lot of things that's very practical for them to actually get jobs. So I don't think anyone can miss which reference desk this is. So even though this student coming up clearly, this reference library already knows a whole lot of things. This is not the Parsons School of Design. So it's, and already she knows things about the likely subjects, the likely characteristics of what a good answer will be for that student and many elements of that aspect of context. And if this were the Parsons School of Design and it would look different, you can imagine how lucky that student is that they are now talking to somebody who has helped several generations, not generations, but several years, decades, perhaps, of students get their MFA in 14 months. So she's going to, or he is going to know a whole lot of things to suggest to that student that the student wouldn't have asked, let's say the student only asks a few questions and doesn't even ask about things that the reference library will know. By the way, you're really going to want to know about this particular database because I know your first course is going to be in this particular aspect of design and you'll be doing really well if I show you these things. It'll all be mirroring many of the things I mentioned about what Samuel Green talks about in his article, but now applied to the Parsons School of Design. So I'm now going to describe to you a framework that we've developed at Credo that allowed us to really define how we wanted Credo to interact with people. And it doesn't mimic the reference interview. It's really a matter of thinking of the query system and it relates to, we call it the user modes of reference. And each of these modes are defined in a certain psychological state that a user might be in. Now this is, it's not uncommon for system designers to have models of users. They tend to have models of users which are, well, we've got one model as a newbie and then we've got the expert and we've got the teenager and then we've got the lawyer. So this model is actually at a finer level of granularity. It basically says that the same person may actually have a different expectation state with regard to certain queries. And they may not actually even be self-aware of which state they're in. So you can imagine a doctor in the morning is prescribing a medication and wants to know if there are any counter indications for that medication. And in the evening, early evening, he's finishing up a paper that he's going to present and he needs some citations for other references. And he has a little bit of time left over and he's told, been told about a great site that has information. He's a ear, nose, throat doctor and he wants to know, what are the 10 things that I had to know about dermatology. And so now he's exploring this whole subject. That same doctor in those three settings have very different expectations in terms of what constitutes a good delivery of information. And most importantly, the issue of the difference between false negatives and false positives. Now, if any of you don't know what false negatives and false positives are, just put a question in the chat and we'll cover that. False positives are very much things that are unexpected, serendipity. And I still remember the first search that I made on what Credo used to be called X-refer plus. And even before I joined X-refer 10 years ago, I got on and tried a concept map, which is one of our exploratory tools and you'll see some versions of that later. My wife teaches English as a second language and for many years, she taught a group of refugees in suburban high school in Boston who were foster children taken from a refugee camp in Kenya. They were from the Southern Sudan and they were from a tribe called the Dinka. And for very obscure reasons, it turned out my father had done some research back in the 30s where the Dinka as a tribe were an important element. So I decided to do a concept map on Dinka. And up came a whole bunch of things about sub-Saharan agriculture about Aboriginal metaphysics, about a whole lot of set of things that would obviously be related to a tribe from the Southern Sudan. And then along came with another node around Doris Day. Doris Day, what could that possibly be about? And I thought, did she achieve, is she a foster parent of a Dinka child or something? And then I drilled down into that particular cluster and there was songs that Doris Day had done with Jimmy Durrani, including the song, Dinka, Dinka Do. So this is an example of a false positive. And if it's presented correctly to a user, that can be completely delightful. If it's presented in an area where you're studying the details of the Dinka tribe and suddenly Doris Day and Inc of Dinka Do come up, it's an extreme annoyance. So the modes of reference was intended to be able to get at the differences between those two user experiences. And we came up with four major categories of modes. And this was useful for us because some of these are ones that we felt that we didn't right now have the full capability to deliver. Detailed bibliographic research projects, as you would normally do for the literature search of a PhD student because they need to be able to have a very detailed and also their tolerance for false positive and particularly false negatives is very, very low. So awareness. So this whole area of research supported something that we do a little bit of, it's not a natural thing for online reference because by the time that something's in an encyclopedia, it's awareness is not going to be an issue. So we now do have some awareness capabilities on some of our topic pages, but it hasn't been really central to us. But the first were the areas that actually extra for plus with X1N, which is this fact finding it's sort of ready reference, a quick answer. You've come across a word in French while reading an English novel and you didn't study French in high school and you just want to get an answer and get back to what you're doing. In a very large percentage of interactions on the computer and in the iPhone when you're driving with your buddies up to on a trip, you know, a lot of it will be cheap. Well, who won the World Cup in 2010? And you know, these kinds of quick answers and you're quickly on your iPhone and you got an answer. You're not source sensitive if you're in quick answer mode. You are if you're in definitive answer. And definitive answer was now you suddenly need to be able to know that it comes from a source that you can stand by. And sometimes that's advice from a librarian. Sometimes it's just it's really in the sense of the user's need. So perhaps you have a bar bet with some buddies and, you know, you know they won't take anything from the globe. So it has to be from the, from the Herald. So we found that we had to do this well. But more important, the area that we really wanted to excel at is in this whole area of discovery. And you're going to see some examples of this. And indeed, the poll that I gave you was, you know, where do students get stuck and need help? And I'll go over some of the answers that we had from this. And unguided exploration is very much like surfing the web. You know, people who like reading encyclopedias. And my guess is about 15 percent of any audience I ask will say that will admit to the fact that they like reading encyclopedias. But the ones who do, what they like about it is this randomness that they don't know where they're going to end up. And it doesn't matter because they're on a journey they're enjoying. Guided exploration has much more of a purpose in mind. And diversion is indeed ways of making the whole thing fun. And one of the ways I found that that was important in reference is I wrote a piece for the opening of the ALA's Guide to Reference. And in doing so, I went back and reviewed the 1902 Guide to Reference. And they had a list of the hundred must-have titles. And when I went through that hundred, there were four or five that I couldn't categorize as anything other than they were meant to make the whole library fun. And I think that any system needs to be able to do that. So we're not going to go over the details of the survey results. But I did want to be able to just share with you a couple of the things that came from those that stood out for me when I read the survey. And some of these are new, they're different over time over what I've seen over the last several years. So a lot of them evaluating resources, knowing where to go next, having a vocabulary with which to instruct their queries, broadening or deepening searches. And one that really stood out many times was that students get stuck when suddenly they're drawn, they have an entry and they go to it and it doesn't get to the full text. It turns out that it was a bibliographic entry or there's a firewall. So they're just frustrated because they're teased. And I like to use this phrase discovery without delivery is a disservice to users. Now librarians may be able to make use of bibliographic materials because they know what they are, they know the value of them, and they know what circumstances they're to be used. But a student with a paper due tomorrow, it doesn't do them any good to show them stuff that they could get my inner library loan if they spent three weeks to do it. So a system really needs to be able to figure out how to avoid that. And Credo has always been about full text. Obviously, we point to other databases, so it's not always the case that we can control the whole experience. So I now want to draw into a little bit of a discussion about what Credo does, what's the secret sauce inside of Credo? One thing we do, sometimes people think we're an ebook vendor. And we actually, what we do with our content when we get it, we have a publishing system that tears apart what comes into us. And we look for content that is actually chunkable, that makes sense in small pieces. And subject encyclopedias are the bread and butter of Credo, and they're certainly matched that requirement exactly. So I'm going to actually limit my discussion specifically to that content type. It's about 80% of our titles. And we strip them apart into their own individual entries, but we extract the human intelligence that's in those subject encyclopedias. And that human intelligence consists of both what keywords have been chosen to be about that subject. So if you have the encyclopedia of gun violence, we have something like that from ABC Clio. And you know, it's going to have a vocabulary of what's related to gun violence that is been chosen by a set of editors. These are the important terms that you need to know if you're going to really understand gun violence in the United States. So that's a human editing curation that's happened, and that we extract that knowledge to drive some of what we do in creating our concept maps. And the other thing we have are the sea also. And if you think about these sea also, those aren't just comments about, they're not just specific to a book. So if you have an encyclopedia of strategy, and you have an entry on strategy, let's say it's encyclopedia on general systems thinking. And so now there's a section on strategy. And so strategy sea game theory. And in the game theory one, they didn't bother saying sea strategy because they thought it was obvious. And being constrained to the limits of what's on a page in their entry, they were only allowed four sea also, and so they thought that we know that the relationship is symmetric. They said, well, sea game theory. So it turns out that these relationships are both symmetric and transitive. And if that doesn't make any sense to you, please ask a question. We'll cover it during the Q and A. But by being both symmetric and transitive, it means that internally, inside of credo, we can create those other relationships. And we can create a relationship between game theory and prisoner's dilemma, all the way back to strategy. Now we don't take it to a complete closure, but you can actually, what we do is a partial transitive closure of this network. And that's what drives a lot of what we see in credo. So now I'm going to show you how that actually can work. So I don't know if you can see the specifics of this particular map, but it's something that somebody was taking an introductory course in evolutionary biology. And one of the things that I learned many years ago in a setting in which we did a lot of work around accelerated learning is that people learn fastest, what they almost know already. So it turns out that I actually already knew what allopatric speciation was. So I sort of might know what simpatric speciation, but somebody just popping into this speciation map might not know what those all are. But let's assume that they end up being intrigued by this question of simpatric speciation. And they're researching this not to do a detailed research, but perhaps to review for a test or for ideation in deciding what paper they're going to write because they have to write a paper and they don't know what topic they're going to write about. So they hit on speciation and that came this concept map. Now we could do this live if we were live on credo, but assume that they just now clicked on the simpatric speciation node. And that will then be, it'll all reconfigure itself. It looks like a lava lamp when it does it. It's really neat. And now simpatric speciation will be at the center. And now there's a whole map around simpatric speciation. And you know, let's say the student noticed evolutionary distance and that's how to like something they might want to learn more about. And they can do a mouseover and you can see the mouse over popped up what evolutionary distance means. So now if you think back to some of the needs of you know students being able to broaden the search or deepen a search, this is a really good example of how to be able to do that. So another example of how credo has done things in terms of some of those elements of context is the literati topping pages. And I know this is really fine print, so it's going to be able feasible for you to see. But when you get into leading on the last page a trial ID that is good until the end of November. So you can get into credo and see these topic pages. And the thing that's really fabulous about topic pages is it deals with the fact that if you're a student at San Jose State and I happen to be at NYU, you know, you have different databases than I do. NYU has 1,800 databases. The chances that an incoming freshman knows which databases are useful for DNA is, you know, is very, very small. In fact, there are a lot of, you know, interesting cases where people have talked about usability studies where the student was looking up, you know, some neuroscience and they went to JSTOR. And when they were asked, well, it was a junior and she was a good student and they asked, well, why did you go to JSTOR? And she said, well, my English professor last semester told me about JSTOR and it was really, really useful. So this is the level of understanding that many, you know, even really good students have and so the whole idea of literati is to be able to say, all right, if I'm looking at DNA, then the databases and further places for study that I'm going to find on this topic page will be things that the librarian at that institution says, these are the right places to go. So, you know, if you were at San Jose State and you looked up Ezra Pound, there's no sense in taking you to the Columbia Poetry Database. It's a great database, a great place to be if you wanted to study Ezra Pound. But San Jose State doesn't have the Poetry Database from Columbia, but NYU does. So if I'm at NYU, I get a different topic page than you do. Even though these topic pages live on the open web, when I click on mine, I get the NYU version and when you click on yours, you get the San Jose State version. So we do a lot of stuff around IP authentication and IP recognition and cookie chasing and a whole lot of other things to give you the most, the best set of topics to go to based on input from your librarian, if you would subscribe, if San Jose State were to subscribe to Credo. And I get one that's based on the input that the NYU library has. So anyway, I'm not going to go over all the details of all of this, but basically this sort of laid out, sort of laid out here, sort of the three major steps in Credo's evolving vision. The first was that an entry in a subject in psychopedia is this little atomic psychomic microcosm that has within it, if you extract the human intelligence that's in it. It is a launch pad into where you would want to go next. And we have been XML, XML was first promulgated in 1998 standard and Credo started in 1999. We have been thoroughly XML right from the beginning. And so we recognize certain entities of people, places, events, works, institutions. And if you take in a particular entry and mark it up for those things, they then can become the launch, the linchpins or launch pads to take you to where you want to, where you might want to find out more information. And this follows very much what a learner's pathway is going to be. And it's not just inside the library, it's also even in resources outside the library. So the second part of our vision is this fact that we can put these topic pages on the open web and use them to entice students into libraries without even their knowledge. I mean, they'll notice it as it happens, but they're not going to have to decide, oh, I think I will go to my library's website and do this query. Our whole vision in this aspect is to have topic pages be very prevalent on the web that people trip across them right when they happen to be curious about something, and then they end up in their library. And the way that we're doing this, and it's a work in process, is both through search engine optimization, but also through partnerships with other players. So an example is EasyGib where, you know, if you cited Wikipedia, it would pop up with a pop-up saying, you know, your professor may not want you to cite Wikipedia. Would you like to see a credo topic page that you can cite? So we're looking at more and more of those kinds of business opportunities that will allow these things. So my microphone says it's about to run out, so hopefully it'll last a little bit longer. So if it suddenly we can't hear me, we're going to have to shift to my computer's microphone. So the third level of this vision is that indeed we can provide a whole set of services built around the reference in psychopedias that focuses on information literacy. So anyway that's the conclusion of my talk about comparing Google and credo. I hope that it was useful and it does now give us a few minutes where I can talk a little bit about against the grain, this op-ed piece I did about filters, and then I'll leave for you a description of our internship program. This question about filters, Clay Scherke, who's a famous media scientist at NYU, talked about how he gave a session about five years ago in which he talked about how when people talk about information overload what they're really saying is that their filters are failing. And he gave the example of how when Gutenberg came out it suddenly was able to produce more books than anyone could read in their lifetime. But it created this odd commonality where the person who's producing the publication has to also know about what's worth publishing. And the reason is because it's very expensive to put together this press. And here's a very elaborate one from 1850s. So if they made if they published the wrong thing and it didn't sell commercially it would be a big mistake. And so if this publishing industry grew up with this combination of knowing how to run the press and knowing how to choose what's worth publishing. So what's happening today is because the publishing cost has gone to zero any one of you later this afternoon can post on a blog and it's available to the entire world and it wouldn't have cost you a dime. So his sense was basically it's not information overload we've always had information overload when it is it's filter failure. So I put this question on on its head and said all right well what's filter success? I don't have any answers to these questions. If you haven't if it spurs your mind please email me i'm at dove my last name at credoreference.com I'd love to keep you engaged in the inquiry and who knows filters maybe people will be buying filters and not just books for their library and then the question was who produces these and is there a seal of approval that says all right this is a well established filter that it behaves in the proper way it allows people to opt in and opt out. So that's my filter spiel that was in the against the grain this month. And finally just a few slides about credo's internship program. They said San Jose states one of the one of the top schools in providing us interns and if you have any friends who've had one of those internships talk to them I think that they they the feedback that we get is that they really find that this has been very special and different from other internships and they're all paid and you'll you'll you won't be paid as nearly what you're worth I assure you because you'll work hard and but you'll learn a lot and we do believe that they all should be paid something and and the way we're able to make them work virtually is through the now available technologies that 10 years ago when I started it at credo this a lot of these things were not in place and we could not have done this kind of virtual internships but you will be working on real customer problems that we have and engage with those customers directly you won't just be doing back office kind of work I mean you will you will do some back office work because we all do but you will really do a lot of meaningful stuff so and there's a little tip from our HR person on how to get hired because so you can apply this anywhere you go it's valuable so and here at the website to check in terms of virtual internships and be there the the trial ID which I promise so that brings us to to 1245 and I believe now we're available for this next 15 minutes absolutely thanks John so much for your presentation and while people are thinking of questions they may want to ask or putting something into the chat box you'd like John to potentially follow up about let me go ahead and follow up on on his last sort of theme of the internships that we've had now with credo for a couple of years and this to some extent came out of an interaction at a conference a couple years ago where our director Dr. Hirsch was I was there and I was a faculty supervisor for the first few now these are all virtual internships you again do not need to be in the Boston area to do one of those so Dr. Frank's supervises those but if anybody's interested you can check out the URL that's John got John has up there and let me remind you that we had an orientation for spring internships last Saturday and that recording is available and we have a second orientation session for internships tomorrow evening 6 p.m. Pacific time so if you are interested in one of these and what John didn't say was that oh two or three or four of the first interns over the first couple of semesters in fact got hired by credo that that's how well the program went and that's how much they enjoyed working with our interns so it did in fact lead very directly to some employment and so now if you're an intern with credo I think you're your internship supervisor at credo is one of our former students one of the people that have actually gone through the process so if somebody has a question you can either click on the mic or put something into the chat box and we'll have John respond soon to have at least one person typing so we'll we'll wait patiently well now and now I don't see an indication of anything oh there they are there's some questions so John can you read the uh are you seeing those questions in the chat box and is your mic working and can you respond to those right I think I'm back on so can you Bill can you hear me now yes yes I think we can hear you now so if you can read the question right in the questions are in the chat box yes I see a question from Ray it says this right does everyone able to read the question already or do I need to no I don't think you need to read it everybody should be able to see the chat box yeah so this is a really good question because one of the things that's been really valuable to us is by having a really brilliant way of extracting the human intelligence of these different subject encyclopedias means that we do not need to have a controlled vocabulary or a or human editors going through and deciding these relationships now we certainly do from a quality perspective sort of search quality we sometimes will fiddle with what comes out just because we think it might be misleading but it's not a standard part of the process the the content speaks for itself now part of that is served by the fact that the reference experience is general and so in a lot of those exploratory modes false positives are perfectly all right when you're in the concept map and you're as I was describing with dink up it you can describe it and you can see that it's a person who's sort of in that mode they're in a they're in a situation where they're going to be you know comfortable with the fact that they're going to be presented with some things that aren't quite right so that means it gives us significant latitude to be able to build at scale thousands of subject encyclopedias and still have a good user experience so if you were building a difference reference system that was very specifically about you know a hospital setting in which case the whole context of the reference was medical information then you'd want to follow a controlled vocabulary around mesh and make sure that things are properly understood that there's a synonym between heart attacks and myocardial infarction and everything else but in a general reference sense that's not been a need for us I sometimes use the expression we're superficial and proud of it because our purpose is to pass somebody to where they ought to go next and not try to answer everything that they need to know so it's part of our empowerment of users is actually recognized that the role of reference is really to make the rest of the library come to life and not try to be the whole library itself so we've avoided these uh overly rigorous you know senses of either of a human process to to edit all of these things or one that would go towards the controlled vocabulary kind of direction Lucy Carroll comment yeah I think indeed a lot of ways to use the concept map to do kind of prejudge these kinds of questions exactly right right that's all those two questions well we have any other questions or comments again if you want to put something in the chat box or if you want to grab the mic please do so and and I'm I'm putting in my email address so you can email me if there's something that a question that occurs to you later or if you just want to add to the conversation I'm glad to follow up with you great thank you John looks like we may have another comment from Lucy there's also the recording and we'll have we'll also do a YouTube version of this and that will go out to everybody when that is done and John I'll get you the URL to the YouTube version when that's prepared so Lucy does have a question great well thank you very everybody oh I'm sorry here's a question who puts together the topic pages there's actually most topic pages are generated by a algorithm that we use we carefully watch over them to make sure that there's not something really egregiously wrong and there are some problem areas where you know if you look at Pluto does it you know mean the planet or does it mean the Greek god or you know there's a lot of areas where there's a lot of homonyms and we try to sometimes disambiguate those we have to do a fine tune but there are some special topic pages which this is one of the things that we have interns do if we do a if we have a literati assignment where we need to bring to life some of the things that are most important to a particular library because it fits with their strategic initiatives and there a handful of topic pages that would be really influential in that experience that we will handcraft the topic pages for those for those settings so and my my recollection is that some of the first interns in fact did that from some requests you got from some some of your academic customers so that was and they found that to be just a phenomenal experience exactly but we have over 10,000 topic pages and a lot of them so most of those are generated automatically but this you know but we really the literati product is really focused on the specific special information literacy needs of a particular institution so we've got hundreds of those but most of them are generated automatically great okay again thank you John for being with us today wonderful presentation thank everybody who was online with us and our next closing session for the fall will be on Monday it's going to be a Monday November 25th that's the Monday prior to Thanksgiving and our speaker will be Virginia Tucker Dr. Virginia Tucker who's also on our faculty and she'll be talking about expert searching and some of the things she found out about what makes an expert searcher with her recent doctoral research so some of that information is online on our website already and we will get out announcements later in November prior to that particular session so thank thanks everyone again and again thank you John very much for your presentation thanks a lot Bill bye