 Hello. I'm Leigh Ann George, the coordinator of the spec survey program at the Association of Research Libraries. And I'd like to thank you for joining us for this spec survey webcast. Today we'll hear about the results of the survey on learning analytics. These results have been published in speckit360, which is freely available at publications.arl.org. Before we begin, I just have a couple of announcements. Some of the presenters have been muted to cut down on background noise, so if you are part of the group today, feel free to speak among yourselves. You won't interrupt. But we do want you to join the conversation by typing questions in the chat box that's in the lower left corner of your screen. At the end of the presentation, we'll read your questions aloud and then the presenters will answer them. This webcast is being recorded and we'll send all registered slides and a link to the recording in about a week. Now let me introduce our seven authors. Michael Perry is head of assessment and planning at Northwestern University Library. Kristen Braney is data services librarian at the University of Wisconsin-Milwaukee. Abigail Gogan is assistant professor of information services and liaison librarian at the Library of Health Sciences at the University of Illinois at Chicago. Andrew Asher is the assessment librarian at Indiana University Bloomington. Kyle Jones is assistant professor in the School of Informatics and Computing at Indiana University Indianapolis. Brooke Robert Shaw isn't able to join us today. She's assistant professor and assessment librarian at Oregon State University Libraries and Press. And Dorothea Salo, also unable to join us, is the faculty associate in the Information School in the College of Letters and Sciences at the University of Wisconsin-Madison. If you want to continue the conversation with us on Twitter, you can use this hashtag that's on the lower right side of your screen. They are all speckit360. Now let me turn the presentation over to Michael. Thank you so much, Leanne. So to begin with and set the stage for what we'll be talking about, what are learning analytics? For this project, we use the common Simon's definition above. Learning analytics are the measurement, collection, analysis, and reporting of data about learners and their context for the purposes of understanding and optimizing learning and the environment in which it occurs. So how does learning analytics differ from the more well-known world of library assessment? So if we think about learning analytics as the aggregation and analysis of student data for those purposes of understanding learning and the context in which it occurs, assessment is actually the evaluation of student learning outcomes using those educational designs. So learning analytics is different from assessment because, one, the data is primarily quantitative. And two, it does not have time to focus on the strategic educational designs but observation of student behaviors, learning, or otherwise, and the relationship to the variety of outcomes. So with those definitions out of the way, what topics do we cover in the survey? The main areas focused on learning analytic initiative participation amongst libraries, library practices around those, library and institutional data sharing, data protections that are put in place for learning analytics data, privacy policies and the practices around with them, procedures and trainings for library staff on the various components of learning analytics initiatives, and then, finally, partnerships between libraries and other entities. So the survey had a 53 respondents of the 125 ARL institutions that was 42%. Questions were not required, so the response rate for individual questions may vary and those will be all highlighted in the respective slides to come. The survey was open from April 30th to June 15th, 2018. So what does the overall learning analytic participation look like? Of the 53 responding libraries, 81% indicated that their institutions were participating in learning analytics projects, which suggests a broad uptick across ARL institutions. Nearly three-quarters of those respondents indicated that they had staff allocated to these projects. So setting the stage aside, I'll now turn it over to Kristen to talk about the findings on library data practices. We're both advancing the slide. Thanks Mike, you're being very helpful. So I'm going to talk a little bit about the practices around learning analytics and learning analytics data in particular. So when it comes down to staff allocated for learning analytics and CSI components of institutions who applied saying they are participating, and of those that are participating, the majority of those actually are using librarians to do the data collection and analysis. So all the responding libraries said that they have staff librarians gathering learning analytics data, and the vast majority of those libraries say that staff librarians are involved in the analysis of that data. The numbers are a little bit smaller for non-librarian staff. 89% of the respondents also had non-librarians gathering the data, but only 64% of respondents had non-librarians analyzing that data. When it comes to the types of data being collected, we see chart breakdown here. We're seeing a lot of information on library instruction being gathered, and this kind of correlates with some of the library staff being in charge of collecting that data. Also seeing quite a bit of information being gathered kind of automatically on research usage. But the most common data collected is research reference consultations and instructions. And it's a little bit interesting to see the breakdown here. The darker green is with identifying information, and the lighter green is without identifying information. So it's really a mixed data being collected with identifying information without identifying information. Now what libraries do with that data is a little bit mixed. So fewer than half of the libraries reported sharing the data with other departments on campus or with the central data warehouse, but another 20% did indicate they were planning to share this information within the next 6 or 12 months. Those numbers don't actually encapsulate the form and format of that data that is being shared, that is something a little bit deeper within the actual results you can go look at within the survey. But most of what we see is that libraries are sharing data with other departments that correlates with collection usage rather than data about patron interaction. And this kind of correlates with what historically we expected we'd be sharing collection information. There are some libraries that are not sharing or not sharing all of their data, and we asked libraries basically why they weren't doing that. The top responses for not contributing data to the institution really center around privacy and confidentiality although lack of resources such as personnel and time also played a very important role in decisions not to share that data. I should define privacy versus confidentiality a little bit more in detail to make sure things are clear. So privacy is kind of keeping information to yourself so we're not choosing not to disclose information such as the patron deciding they don't want to disclose information versus confidentiality as the library has the data but we hold it confidential and it's known but it's kept within the library. All right, next thing we want to talk about is library data collection. One thing that's been interesting is that in response to learning analytics it seems that libraries are actually collecting more information. So the majority of libraries indicated this is 91% that in response to institutional learning analytics efforts they're actually collecting the same or more data with personal identifiers than they had previously. So there's some indication that the amount of data that the libraries are collecting is going up with these new efforts to do learning analytics. Despite that increase only about 50% felt that library data was important to campus level initiatives. And it's actually a really interesting breakdown graph here and we asked two different questions. How important do you think library data is to institutional learning analytics? And how important do you think library it is for the library administration to participate in learning analytics? So the green, the dark green is very important, the light green is not important, and the middle green is important. So you're seeing a slight difference in how libraries are judging the value of analytics because when you ask libraries about the importance of library data to the institution in general, maybe a little over half are deeming it important or very important, but when you ask about the importance of learning analytics to library administration all of a sudden you're looking at the majority of libraries saying it's important or very important to the library administration to participate in learning analytics. So we found this result to be pretty interesting that there seemed to be a difference between the perceived value of the library data within learning analytics versus the library data to the administration within the library. I'm going to wrap up my topics about data in particular. I could talk about data all day. I'm a data librarian, but I want to talk about data protection because this is a really interesting topic and one that comes up a lot when you start talking about learning analytics. So we asked libraries to go through some information about how they're protecting that data and that data protection encompasses quite a lot of different factors. So one thing we asked about, for example, was are you anonymizing your data and how are you doing that? Of the 50-some institutions that responded, only 15 described a technique for anonymizing their data. So if they were collecting data with identifiers and then they subsequently anonymized that, they had some description of that. A lot of those actually said that they received de-identified data from the Office of Institutional Research. So when they received learning analytics data from the university, it was already de-identified. There wasn't a lot of really in-depth anonymization techniques that were being described. Another data protection that is often put in place is to limit the retention of sensitive information. And it's interesting that about 38% of the libraries report having a records management schedule or a policy that controls the retention of learning analytics data. That breaks down a little bit further that nine libraries that didn't have a retention schedule or a policy report that they planned to hold onto learning analytics data indefinitely. There's really a wide variety of practices with respect to how long this data is being retained. On the data management side, if you're interested in that like me, only two libraries reported having a data management plan for learning analytics data. To give a little bit broader breakdown of some of the protections that libraries are putting into place, we have this graph here. The top things that libraries are doing to protect learning analytics data is limiting staff access to the raw data which might include identifiers. After that, they're removing those direct identifiers from the data. For example, removing a student ID so you can't correlate library checkout information with a particular student. And limiting the scope of data collection to make sure we're not collecting too much data on any particular student. So there's quite a variety of data protections that are being put into place. Some libraries are doing more. Some libraries are doing less. But the top things that people are doing, libraries are doing are to limit access to the raw data, remove those direct identifiers, and to limit the scope of data collection. And with that, Abigail is going to dig a little bit more into privacy policies and practices. So Abigail, there's a slide for you and it's all yours. All right. Thank you, Kristen. So to address the back half of the survey that we sent out, we wanted to know what are people gathering and what are people doing with that data. But we also wanted to know some of the infrastructure that was available for staff, for students who are engaged with these learning analytics. So we wanted to ask some of those infrastructure questions and see where best practices were and what we could find out. So where we started with this was privacy policies and practices. And as you can see, 90% of the libraries who responded indicated that they had some sort of privacy policy at their institution but a much smaller number, only 62%. And that number is not right. Kristen, do we know what the other number is? Have a separate library policy. I'll have that on a chart in a second. We asked about how often those privacy policies were updated and whether that had been updated, whether those privacy policies had been updated regarding learning analytics. And it was really fascinating to see that there was no consistency when those policies were reviewed, how frequently, and only seven libraries told us that they had specifically updated for learning analytics. Abigail, those numbers are correct. I'm the data person. So does your institution have a privacy policy? 90% said yes. Thank you. Separate from the institution's policy, 62% said yes. But 50 people answered. Correct. The ends are how many people answered that question? Okay. Thank you. So when we asked what was in those privacy policies, so of the 62% who said they had a library privacy policy, what did those refer to? And as you can imagine, the primary thing that they referred back to was that university policy and also then the state policies on library records. A number of them also, so about a quarter of respondents said that they pointed back to the ALA Code of Ethics and then other terms of services. Interestingly, this was right about the time of GDPR. We did not see that come up particularly. Another question that we asked about was about informed consent and how learning analytics projects were reviewed within the library. 42% of the libraries said they informed students about library learning analytics initiatives. However, only four of those libraries said there was a mechanism for their students to opt in. So while there may be telling people that we're gathering this information, there wasn't a specific opt in. We also saw that the majority of our respondents thought IRB approval, and 60% of the libraries indicated that they reviewed FERPA with their staff members when they are addressing learning analytics work. We thought the informed consent was probably the most interesting number out of those, and this chart reflects what it is that we were finding there. So of the 43 respondents to that particular question, 18 informed students that there were learning analytics happening, that we were gathering this information and doing something with it. So 25 said they do not inform students in any way. And of the 18 who said they informed the students, four allowed for opt in, five allowed for opt out, and six said that they neither provide an opt in or an opt out method. So there were only five respondents total who gave students a way to opt out of participating in learning analytics. Now in terms of procedures this builds out a little bit further on IRB as well as the FERPA questions that we had already asked. And we asked about procedures within the library. Were there internal staff guidelines and documentation? And that was only available at 25% of the libraries who responded to that question. Further only 33% of libraries had any kind of process for handling external requests for campus entities. So before Kristen said it's very common for us to think about sharing collections data, and that was where we saw the most data going from the library to a department but there was no particular process at 26% of the libraries for handling those external requests if they asked for that data. In terms of the training that library staff involved with learning analytics, it usually had to do with using specific tools and then IRB and FERPA requirements. City training came up a lot. These were the breakdown of the types of training that library staff used. As you can see the focus was mostly on tools and IRB requirements. Here's how to gather the data. Here's how to use spring share or another specific tool. And then it goes further down. You can see at the bottom there was eight respondents who said data cleaning, nine with anonymization practices, seven respondents told us there was no training. So we didn't track particularly to see what those particular institutions were gathering so that will be an interesting further question in the data what they're actually doing but they're not offering any training to their staff about library learning analytics. And then finally we asked about partnerships. We wanted to know what was happening both at the campus level and then also at the professional level. And almost 40% nearly half of the respondents for that question said they are participating in some sort of learning analytics initiative alongside other campus units. And that raises an interesting question back to the value chart that Kristen pointed to earlier. When we look at it on a professional level, nearly a third of respondents said they were working with Consorcia, Goela, the Greater Western Library Alliance Success Initiative, and the UNISN project both came up specifically as Consorcia where library learning analytics are being gathered on a regional or national level. And I will turn it back to Michael to wrap us up. Thank you so much Abigail. So having looked through all of this data and analyzed the response to the surveys the group came up with some recommendations for libraries about how to approach learning analytics and learning analytics data. First among those, libraries should put in place a schedule for reviewing and developing privacy and data management policies. The process should be handled by an informed dedicated committee office or individual which would be indicated on the policy. And it's really important policies should be written in a clear, concise understandable language so that students actually understand what the policy is outlining. Wherever possible, actual systems and data types should be identified in the policies so that they get a very clear picture about what data is being captured and used. And policies should include a revision history approval process, last review date, as well as contact information for that individual or committee that is handling that work. Wherever appropriate policies should link back to governing documents such as university policies, state and federal laws, and ALA ethics. And the one thing, and this is something that Kristen talked about that I think is vitally important there, is really developing internal data management policies for libraries learning analytics projects. That is the service that many libraries offer for their faculty so it's something that we should have the capacity to do in-house. Second, libraries should explain training on data handling best practices, really that go beyond institutional FERPA and IRB training and requirements. Library staff would most benefit from training on underutilized data protection practices identified in the survey's results such as technical protections like encryption, storage and transit, processes for data minimization including limited data collection and retention times, and anonymization strategies. Library should commit to protecting privacy of their information about their users and their information habits. And such commitments should be applied to the user data that they keep and share. Third, as many projects are perceived to be for internal use only the institutional review board may not be contacted even when the data are subsequently used for research. Similarly, many IRBs do not see the data already collected as carrying potential for harm. Libraries should develop best practices for assessing the ethical and personal privacy risks to students internally. Rather than relying on IRBs regardless of whether they have immediate plans for disseminating the findings of their work. The changing nature of learning analytics data means a lot of those policies that are laid out by IRBs don't really fully bring to bear some of the concerns that might be raised. Fourth and final recommendation, libraries should be more transparent with their students about learning analytics projects. Only one respondent provided a clear document outlining learning analytics projects that were going on in the institution. And it was even unclear if that document was actually publicly available to students. This transparency includes engaging with students to inform them about what data is collected and how it's being used. And really getting a sense of turning them into partners on any potential projects that they are being included in. Thank you Michael and Kristen and Abigail. And now we welcome your questions participants. Join that conversation by typing your questions in the chat box in the lower left corner of your screen. And we will read the questions allowed before the presenters answer them. And while you're typing, I'll get you started here. So what was the most surprising finding from the survey? I'll go ahead and get started. I think one of the things that jumped out me the most that I've been thinking about kind of recurrently was in the types of data that were being collected. And it was the security systems, namely security cameras and CCTV things, of which I believe it was either 23 or 25 people identified that they were collecting that data and using it for learning analytics, which was kind of surprising to me. The thing that really stuck with me though in thinking about that was one of them tagged that as not identified data, which is kind of interesting as you think of video of someone doing something in the space. There's any number of efforts to underway in different industries that look at trying to re-identify people based on data, video data. So it was really interesting not only that we were collecting and using that data, but we thought just because we didn't have in place a way of connecting that to an identified person, it should be obvious that really video of someone is identifiable. I think for me, speaking as a data person, I think one of the things that was most surprising is how all over the place the retention policies were. I kind of had a sense of people doing more or less data protections and that made sense with what I see with researchers, but I guess it's because I'm at a public institution which has record management schedules for everything. So I was surprised that it was under half had retention policy for this type of data, and particularly surprising was the number of respondents who say that they were holding onto this data indefinitely. And that raised a few concerns with me about how to protect that data if it's being held on to for all time. So we do have a couple of questions coming in, so I'll go ahead. This is Kyle Jones. By the way, I'll go ahead and read off the first. So the first is from Sarah Lippincott, and she says I'm really surprised by the finding that so few librarians think that the data they collect are relevant to campus learning analytics projects. This seems to imply that many respondents don't believe their library's activities and services contribute to student learning or success. Is there another explanation? I think I can take that one. This is Abigail. I don't know that it is that they don't believe that they're not contributing to learning or student success. I think that they don't see that the quantification is useful at the campus level. I would be hesitant to assign a lack of value of library services to whether or not hash marks of who's walking into the building or counts of who's walking into the building is useful at the campus level. So I think they're two different questions, and I'm not sure that I'd have to go back and look at the data. We didn't really have the opportunity to go into why they felt it wasn't that. That would be a very interesting question to dive deeper into. So I can follow up on that for a second. So I've been really interested in looking through the qualitative results, looking at people's comments on various questions. One of the things that really stuck out to me is kind of a theme is that people express that this is very new work. This is nascent work. They're trying to understand better what the capacities and capabilities are. So it may not be that they don't think it would be valuable. They just don't know what the first steps would be to interacting at a larger institutional level. And this is Michael. The other thing that I was thinking about was I think in a matter of just trying to execute these projects, there's probably bigger systems that should get more bang for your buck out of leveraging first. Namely, whatever your learning platform is be it Canvas, Blackboard or something else. The wealth of data that's available is there ends up being the primary driver for a lot of these projects. Whereas library data tends to be very siloed between individual systems and in often different formats. So I think a lot of places are starting with that more centralized, larger data set. And then beginning to look at, well, okay, if we have that, how can you pull in other things? Going on to another question here by Roxy and Selberg. The question is, did any of the respondents indicate any positive results from any of the learning analytics work that they were doing? I'm not sure that was really the focus of our survey in terms of what people were actually doing with the results. We were more trying to figure out are people participating in learning analytics and some of the considerations around privacy and data workflow. So I think that maybe Kyle has a different answer having gone more into the qualitative pretext response of that. But I don't think our survey was necessarily designed to answer what people were doing with the results of their learning analytics endeavors. Yeah, I would second that, Kristen. I didn't see anything in qualitative data that suggests that people were commenting on the efficacy of these projects. They were more talking about the processes, the practices, the documentation that forms them. So it would require different questions to get at those types of results. And the following question kind of connects to that last one. And it is, do any libraries maintain documentation of the studies or research being conducted on a regular basis? And again, for my understanding or my review of our work, that's not a question that we asked about. However, if that's something that you are interested in, I know from my own research I've been really informed by looking at the OCLC database on the value of academic libraries research. So it's a very good data visualization tool for putting down what type of, what I would consider the learning analytics type of work that's been going on in academic libraries. And just generally to add on with that, I didn't get the impression that they were documenting the research being done. There may be one or two exemplars who are tracking that, but I didn't get the impression when people were talking about where the data gets used that there's good either internal or external. This is where we use this data because of the low number of data management plans and retention and that kind of documentation that we did find. Yes, we did only have one person that did actually provide a document that outlined learning analytics projects in that sort of holistic way. I'll be honest, it's been a while since I looked at that specific document because it was attached not within the survey results itself. So I don't even remember the detail that it went into, but there was only one person that had anything even resembling that that was provided. So moving on to another question or last question that we currently have, Catherine says that the important analytics to me, a library and educator have to do with to what extent students are applying what they learned in the library to their lives and work. For this, we use the Kirkpatrick model of training evaluation. Do other librarians use this model? If not, what models of evaluation are they using? How well do they work? Do you have any info on this? In a model like this, be useful for this broader question of the value of the library to students' success. So we didn't ask that. I know I'm not using the Kirkpatrick model, so I will speak for myself that I'm not using that. So we did not ask about application. I think that would be a really interesting follow-on or something else to look at. But this was much more focused on I think what we're seeing right now is a much bigger focus on quantitative capture of usage primarily. Like, do they have a reference consult? Not what did they learn and apply and do out of that reference consult, but did they have one? Or ask a question at the reference desk. I think we're still in that nascent phase where that's more of the questions that we're asking are these things we can pick off pretty easily rather than necessarily being able to look at the application further down. Yeah, and I would echo back to what Mike said at the beginning of the presentation about the difference between learning analytics and assessment, because we really didn't get into assessment that really deep are we being effective versus learning analytics, which is more the quantifying what we do in libraries, which was particularly what this survey is asking about. I suspect there's other literature out there that goes more into these deeper measures of assessment using various models. And this survey is really looking at I love the word nascent. I want to start using that all the time when I talk about library learning analytics. It's kind of nascent growth that we have with quantitative assessment kind of across the board what data do we have, how can we use it, what does it show and showing that that's really kind of a new area of growth in the library. So I have a question for you to talk about this nascent project. For our listeners who maybe have yet to begin learning analytics projects or just getting started with LA, what do you think should be their first steps for ethical practices and protecting privacy? That is a lovely question. Yeah, that's a very, very good question. So I have an answer from kind of the practical side of things because I think about data protections a lot and somebody else can answer kind of the ethical ramifications. But I think really having a considered approach to all this data collection and kind of thinking through things, what do you really want to know, what did you do need to answer that question, trying to think about do you need that to be identified at the individual level or not. And if you do need individually identified data, what protections you're going to put into place and how long you're going to hold onto that data before you hopefully delete it. So I think kind of a more measured approach is what I personally advocate for kind of thinking through the privacy and the data protection implications as we go along. Because I think it's very easy to say oh my gosh we have all this data and go for stuff and then realize you're doing things that might have privacy implications. But if you think about privacy through kind of all those steps and how that plays out in terms of what you're collecting and how long you're keeping it, that's the approach that I would advocate for kind of as a data protection advocate. My colleagues might have other answers. Go ahead. I think Kristen is setting me up here since we butt heads on whether or not we should be practical about these questions or a little more philosophical about them and I tend to take the philosophical angle. And I guess what I would ask of all of you would be to consider the consequences of participating in this space in the first place. What social structures are you supporting what power dynamics are you enabling in what ways are you propping up the quantification of student life in ways that don't need to be quantified or can have a lot of other harms on student life besides privacy but things like autonomy and a student's ability to pursue his or her interests according to insertional values. Those are the questions that I'm a bit more concerned with though I definitely appreciate the practical approach in having kind of privacy embedded in the practice of doing learning analytics. I think that is the way to do it instead of thinking about it as an afterthought. Go ahead. And I think to follow up on what you were just saying, one of the important things for me was that fourth recommendation, be transparent about what you're doing. I know that students understand what you're doing, what you think the perceived benefit of doing it is going to be, and really try to make them partners in this work. It was a conversation that I recently had with our campus's head of enterprise systems and analytics about his conversations around this because he's very interested in the notion of what student perceptions of privacy are and how those might be changing. And when he's talked to people he's like the most frequent response I get from people is just do you think this is going to help me? And can you explain how this is going to help me? So I think in thinking through those things that Kyle pointed out, really think through too what the actual benefits for students are going to be. And I think you can make an informed decision about is the potential benefit going to outweigh concerns that you have. And then in making sure that the whole process is transparent to students they can get information about that. And I personally think can opt out if they so choose. I think we can offer a system that I think we can all feel a little bit better about. I will point my favorite question which is what happens if someone who doesn't have good intent gets their hands on this data? We tend to approach it from how will this help our students? What will this do? I like to play the devil's advocate and say okay well if we're tracking all of these things like that video tracking that Michael mentioned before what's the worst? What if we have someone misuse that data to stalk someone or something like that? What are the worst case scenarios because we need to be thinking about those now? Thank you presenters and thank you all for joining us today to discuss the results of this learning analytics survey. And I will remind you once again that you will receive links to the slides and recording in the next week.