 Good afternoon. My name is Sam Van Horn. I work at the University of Iowa. I am the ITS Assessment Director and just to let you know I cannot see anybody on this side of the room so if you have a question you will need to jump up and down and I will try to see you or someone on this side of the room please point out if someone over there has a question. I do feel like I'm kind of a star here if just because I have many lights in my eyes. My co-authors are Jane Russell from my office as well as Professor Kathy Shoe from the College of Education. I would also like to reach out and say and welcome the online audience today and I look forward to engaging you as well throughout today's presentation. If you're using social media please use the please use direct things to me as well at learning places. Obviously my phone's in my pocket. I might I'm going to be not using my phone during my presentation but I would like to be able to participate in the conversation and whatever people may be talking about and obviously you know to use the regular EduCause hashtag. So I'm here to talk about assessment using multiple sources of data to support students who are using educational technology and I just want to kind of begin by saying how did we get into using electronic textbooks and I just want to say that what happened was we got a bunch of free electronic textbooks. The University of Iowa participated in a pilot project about two years ago in which students were given a free e-textbook to use that was accessible through the course management system and that we would also be able to access the data that was collected in the database. So it provided a very good opportunity for us to look at how students engage with e-textbooks and so we decided to take advantage of this opportunity to design a study and look at these things. Just to say really briefly I don't want to go into too much detail here but I do want to talk about just a basic review of the most I guess often researched areas of electronic textbooks and I will even say admit now that the term electronic textbook is becoming somewhat of a misnomer. Many of you I'm sure have been walking through the exhibit hall. You have seen products from companies like Pearson or McGraw-Hill and seen that these actually contain a lot of different types of content now and don't really look like books. With respect to learning outcomes there's been a variety of research conducted. In general research conducted in higher education settings in which the outcome variable say is the final grade in a course. The outcomes generally have shown no significant difference between students who use a paper textbook and students who use an e-textbook in the course. However there's been more recent research about how students who use an e-book or an e-textbook have learned less in conditions more laboratory like conditions in which people have used e-textbooks. Probably most often researched is what students prefer and and definitely the research here is split in terms of there have been studies that have shown that people strongly prefer e-textbooks for reasons such as they have good access to it provides a search tool and there's also studies that have shown that people prefer paper textbooks for reasons that they can always access it for example. And more recently researchers have been trying to look at well what are the factors associated with students decision to adopt e-textbooks and a very interesting paper by Stone and Baker Eveless suggests that perceived usefulness of the tools as well as prior satisfaction using e-textbooks is very important when considering if I'm adopting an electronic textbook whether actually students may be on board with using it. And those preferences are mediated in addition to price. And now that electronic textbooks now that they contain they provide possibilities for interaction these are not usually these are not just scanned PDFs anymore people have begun to look at well how do students interact with their instructor with others in these interactive platforms. One relevant study by Kuhler and Doolin found that while the instructor had used the tools extensively the students most of the students had not engaged with using the learning tools at all. So primarily researchers have looked at whether students are satisfied after they're using an e-textbook whether they've learned better after using an e-textbook but there have not been many detailed use examinations of how people are using e-textbooks throughout a term. And so that's what we really decided to do and just to kind of just provide some brief background about what this e-textbook was is that it was not optimized to use on a mobile device was optimized to use on a laptop like mine or a personal computer. It allowed for online and offline reading it included all the basic markup tools you might find such as the ability to highlight ad notes use annotations submit questions to the instructor and bookmark pages for easy for easy access later. It also included an option for sharing markup tools with others. A search feature and then a scrubber bar a bar at the bottom of the screen that enables people to to quickly slide through and look for relevant pages. Now one thing we wanted to be sure is that when we looked at how people used e-textbooks we wanted to be sure that we were looking at the main textbook for the course. So a precondition for participating in the in the pilot and then obviously in the study was that the e-textbook was the e-textbook was the major textbook and that they assigned at least half of it for the course. From the students we collected a baseline survey as well as 14 weekly reading journals that we sent to them electronically as well as a comprehensive post survey at the end of the semester. From the e-textbook system we collected a large database of information about their reading as well as their usage of the different markup tools and then we collected information about the demographic information and information about the prior learning outcomes and outcomes in the course from our university registrar. Our research study was approved by human subjects. So I feel like I've described a basic study and but if we stop to think I've just basically said that I've collected information about how about reading from the vendor that they're going to tell us who read and this information is going to say well people read or did not read the e-textbook but I want you to just think about this briefly. What does it mean to have read a page in an e-textbook? What would you consider the main criterion to be to say this page has been read and this page has not been read? And so I would like to engage you in some audience participation and for this I actually do need volunteers and you don't have to come up and be on camera if you don't want to. You can do the activity here but I do need two volunteers for a brief activity. It's painless. I trust you. Could I have two volunteers please? Okay I have two volunteers could you please come forward? I have to stay on camera. I'm afraid if I walk off camera people will turn me off. I'm going to hand you a paper to I want to see people see this is a paper textbook. I brought this textbook from Iowa I want you to know that and I almost incurred an extra baggage fee and this is my iPad are you comfortable using this? This is EDUCAUSE you should be. I have directions on the slide and the viewers and the online audience I also have posted directions for you to use so that you can access the e-book version this is a textbook a biology textbook and what I would like both of you to do now is I would like you to navigate and find page 126 and I would like you to read both of you to read page 126 and for those of you in the audience at this time do any of you want to I guess hazard a guess about how long it might take to read a page in a comprehensive biology textbook you can feel free to step up to the mic and and place your bet or if you say it out loud I will repeat it for the I will repeat it for the online audience 15 minutes is one guess I believe my two people are still reading the page is 126 it is in section 7.1 any other guesses six minutes people are learning biology right now so I have 15 minutes six minutes have you been able to find the page on the on the iPad you that paper textbook the paper textbook reader has found the page okay so would you like to just try reading it okay I'll give you just a couple more minutes to read here in the online audience does anybody hazard a guess or is anybody reading the e-text book themselves was anybody want to hazard a guess about how long it may take to read so I'm actually going to stop you right now just to ask you just the paper textbook user how long have you been reading do you think a couple minutes three minutes okay are you close to being finished you're at the bottom of the first column so two thirds okay so what I would like to say is that is that the e-text book vendor in which that we worked with said that a page in the system was kind of as read when it was viewed for 10 seconds so this was just to illustrate that a 10-second read in a standard in a standard textbook if you bring that to me I'll hold it up and I will release you from your from your from your duties yes so one so it's a very substantial page in a textbook and you can see that these that after just several minutes that it was only half of it so what this is sort of what I want to kind of talk about here in a presentation today is that when we look at analytics some of these it's such a very hot topic that we you know we collect measures about reading but do we interrogate what these numbers actually mean what are the assumptions behind these numbers and do the and are these assumptions realistic clearly a 10 seconds is not enough to read a page in a in a standard textbook and I don't think this biology textbook is is is out of the is out of the ordinary what I would like to now just ask I'm going to ask a couple questions so just just for a few minutes and and just just and people other people could people that were reading and could you focus on what you were reading did you look at any other pages did you look at the page across did it take longer than you thought it would did anybody with the iPad user did you have what problems did you have or successes you have to speak into the microphone I'm sorry yes so I'm not a great test case for that but the question I had earlier was to how do you how did whoever determined the 10 second rule for having read the page where did that come from because we did some of that kind of analysis too and generally speaking I didn't have any trouble I I read on the iPad all the time I didn't have any particular okay thank you to answer your question our account representative did not know where that number came from either the number was known by the database developer other comments okay so what does it mean so what does it mean to have read obviously that 10 seconds is too is too low of an estimate and what I want to talk to you about now is this I'm going to talk about briefly about what we found in our study in our study in which we examined people that used e-textbooks we included a control group a group of students who used a paper textbook in a matched course and what I'm going to do now is basically summarize the findings by looking at the basic findings of that study before I start to look at how we look at analytics data and more subjective measures in concert with each other so we had 270 individual students in in our in our research study they were primarily female and they were mostly students who were juniors and seniors that the courses that participated in the pilot were primarily courses more advanced courses in the discipline so so we so our sample was is definitely not representative of our university as a whole what did we learn from giving these surveys in these and these reading journals so what did we learn from subjective measures that we that we provided or that we administered during the study well what we found was that even though students had a free e-textbook about 16 percent of them had purchased the paper copy of the textbook and a on a small number had said that they were planning to buy that so a pretty sizable number we're still committed to using the paper paper textbook and I'll talk more about talk more about that group later what we found when we looked at I'm going to show you some I'm going to show you some plots now and what I'm going to show you these plots they are good visual representations of statistical procedures that I don't really want to go into in depth but if you're if you are a statistician please come talk to me essentially what we found was that throughout the semester after controlling for the courses that there are in students who use the paper textbook had greater average satisfaction using with their access to the paper textbook then students satisfaction with accessing the e-textbook all throughout the semester we learned that from their reading journals in addition what we found was that students and these these differences across time are significant statistically significant we also found that students who use the paper textbook reported reading more so this graph shows you number of hours reported reading for all the weekly reading journals they reported that they read more and this difference was significant across time and in the semester at the end of the semester obviously you might guess by seeing that e-textbook users had lower satisfaction with accessing it lower reading times students in the paper group felt that the e-textbook were more likely to agree that the e-textbook I'm sorry that the paper textbook that the ability to highlight help them to learn that the ability to use the index was more useful for their learning and and that overall the paper textbook was more useful for their learning so what we found in general was that students in the paper group used their textbook more and were more pleased with the paper textbook so in order to talk about that now the context of the current information where we start to look at the analytics well what does the analytics now focusing more specifically now just on the e-textbook group what do we learn then if we gather this information and we work to say we can we can build these measures and now at the end of the semester we have a ginormous amount of it's essentially like big excel spreadsheet that that then must be made sense of and so what i want to kind of illustrate to you is how we did this in the sense that this calendar shows you that we we took all of the analytics data that we received from the vendor and and basically it has a time stamp for any type of interaction like i said the page being read in 10 seconds you know hint hint and then we matched that reading data with the survey that was administered the week afterwards so you can see we we administer a reading journal on the sunday or the monday we ask students to tell us could you access your e-textbook how much time did you spend reading your e-textbook how satisfied were you with accessing your e-textbook and so this is how we join that information we can total up the number of pages that were recorded being read in the e-textbook and we can join that with what students said that they did in the e-textbook and does it match and think and think about that what do you expect do you expect this information to match someone said no someone is someone someone's pretty confident there and so so we will talk about that and because what i want to talk about is you know we live in the we live in the era of we're living in the era of big data it feels like an era just a few years i suppose that big data is the truth big data contains so much information that big data by itself can tell us everything we need to know about how our students might be interacting with these things but i want to talk to you about when it may not indeed be the truth and i want to talk to you briefly about a couple of things and one of them is zero what is zero zero is nothing um when we receive a data set of analytics from a vendor and i don't think it's that different for different vendors what does it contain it contains the measures of usage of people that have done things in the system what does it not contain it does not contain information about people that have not used specific information the system so the value of zero is actually a very important decision when looking at analytics and i'm going to show you some information about we had to make that decision where we look our first inclination was to look at information that we received from the cist from the vendor as it was as a measure of how people interacted with their e-text books but then when we realized we actually had a conversation with the vendor about people were missing from the system and we had a conversation for about this these people they were in the class they're not there where were they and it took a while us for us to actually accept they actually never read the e-text book our assumption was being free that people would all be there they would all be have a record in there but that wasn't true and then i also want to talk about what does what can analytics tell us about does it what does it tell us or not tell us about the richness of reading behaviors that we also may want to know about so what this info what this what this shows you is that if we when we take the analytics information as it was and if we calculate some descriptive statistics i'm going to specifically focus on the median there on the far right of the slide and the median is that is the midpoint is the it's the 50th percentile that information you can see here that while no one read their e-text book offline we see that on average at least 50 percent of the people obviously read their e-text book and 50 percent of the people used the markup tools in the e-text book but what this table shows you is that if we go in and this takes a little bit of you know this takes a wrench and some tools but if we go in and then i say if we know all these people were actually in these courses and if we add them in the system and if we add in the zeros what we actually don't have is we actually don't have missing data which is a great thing from a statistics perspective but we actually can say specifically that there was where it says the median is undefined that less than half of the people actually use that tool so what so when we actually look at specific uses when we actually count the people that have not used the tool we actually see that this information is a lot less so it's very important the the concept of zero is actually very important i would argue in looking at at analytics analytics is also quite messy right so here are plots for the for the for all of the e-text book courses the mean number of pages read at that time point for weeks three six nine and twelve throughout the semester it's you know so at what analytics is also it's also difficult it can be difficult to interpret maybe we see we see one course that's much higher than the rest we do see a general downward trend where most of the lines are converging closer to zero at at the end of the semester but we also look at we also i'm sorry we also looked at individual reading data so what we all we could look at how much students said that they read in the e-text book for i mean for the duration how many hours they spent and we also could measure the number of pages that were actually read for the 10 seconds during that period that we asked them about so what do you expect to see if someone reports that they read for one hour and another person and they say well i read for five hours do you what do you expect to see and actually this is a question i'm asking the audience here if someone who reads reads reads for do you expect to see five times the number of pages read so someone who reads for an hour read 20 pages someone who reads reads for five hours did they read a hundred pages are there any could you approach a microphone please necessarily because it depends on the subject the students prior experience with the subject how fast the student can read what kind of distracting environment they were in etc etc so you can get a general sense if you have enough data points you may be able to plot out a line but with just a few especially with the small number of students that you have in your sample here whether you can get any kind of plot it's questionable okay so maybe not other ideas other thoughts would we see five times the number of pages read thank you i'd expect you to see less than five times the number of pages read okay and why why would you expect to see less well i think there's more opportunity for distraction people might linger over something they took them five hours because they read slower okay there are a lot of possibilities also 10 seconds of page they could read 1800 pages in five hours that's probably more than the whole textbook right i think you yes that is total number of pages read so that the number of pages i'm showing you is not unique pages it's so there could be multiple page there could be multiple instances of the same page yes so what so what we did then was that we actually plotted um we use a procedure to look at what we would expect if we could plot the number of uh the number of pages read as a function of what's as a function of what the of the duration that students said that they read and what we find actually is an interesting relationship what we find is that there's a small increase between that zero to one hour so if you're looking at that x axis you're seeing that small increase for the if there's a difference there yes there's a there's an increase the number of pages read but then that line stays completely flat uh until about six or seven hours when it starts to to rise up again and so interestingly when students we might expect from when we see the subjective measures in our study we thought well these larger readers they must be seeing more pages they must be consuming a lot of content perhaps this speaks to the idea that they were studying that they were spending more time lingering over the content and we're actually looking at that we're actually pairing qualitative information to look at what they reported about the richness of their reading with some of these people who's who reported reading for uh for longer durations there now interestingly i also told you that or the early part of the session that we collect information about who bought a paper textbook at the beginning of the term and this line is showing you the students that did not report buying any uh a paper textbook but that we find is a difference and that the students who the students who reported buying a paper textbook read throughout the semester were more were read less uh than the people who read and um who people who never purchased a paper textbook so one of the important things we find from subjective measures early in the semester with respect to e-textbooks is that those early attitudes are very important in assessing in terms of if i have a class who is more likely to be more likely to engage who is more likely to abandon this this um this uh this e-textbook and what we found was that these early subjective measures which are much more accessible to us than analytics data can be very important for decision making like that um what we also found was that uh an increasing number of students who said that they a lot of people sit by then this semester said they were reading their e-textbook but i don't know what they were reading but it was not their e-textbook so um so so these are people that said that they read the e-textbook but they actually did not and i actually can't tell you what they read and maybe they were lying about what they read uh and so uh what were the and so what were they doing there uh speaks to the idea that perhaps there was more abandonment of the e-textbook as time went on but those subjective measures correlating those with the information that we got from the data set shows that uh increasingly students what students said they were reading was not actually what we were measuring um in the e-textbook so i've talked to you about the main findings of our study and how we've put together different information putting together the information from the reading journals that we gathered along with the database of e-textbook usage and chat i want to talk now kind of briefly about some of these challenges of working with analytics information um because we went in this uh gung ho thinking we would have a large amount of data and what we found was that the people that we worked with in the company didn't understand the data as much like we didn't understand it like i like the example i gave you about the number of pages read or what a red page did for we all didn't know these things and we knew the essentially we always there was always a database person that i wasn't allowed to talk to that the question we get filtered back to and the answer would come back to me usage of users have different identifiers so in our study students participated more than one time because they ended up being in several courses that were using an e-textbook we gave them a different subject identifier but the e-textbook vendor gave them one and so the actual data management of trying to merge information that you collect with a separate identifier is actually not a trivial task and it does require pretty significant data management skills that we found that that you actually have to do and what actually the perhaps the biggest lesson that we learned was that we thought we would have this very rich database of e-textbook tools to examine it would let us look at how people interacted with the content who is more likely to use these tools faster who is more likely to work was that associated with any learning outcomes that we could measure and what we found was that we had such little incidents of tool usage that it actually doesn't give us enough power so we actually could not find in our study any ability to say anything about anything related to students measures measures of their prior learning so we couldn't talk about maybe lower performing students were more likely to use these tools or not demographic information and what we found was that because the instructors primarily had used this e-textbook as a substitute paper text all but one had never used an e-textbook before all but one used the paper textbook themselves and all but one never scaffolded the adoption of these of these e-textbook tools so the assumption that we had at the beginning that there would that this usage would naturally occur we found was was basically erroneous and that if I was doing this again and before I was going to dive into this it would be important to understand is there something happening in the instructional design that's going to promote this usage that would then make it more meaningful to look at and whether there'd be more actual interaction that would be interesting to look at now I know I'm not the only one this is my experience working with analytics type of information other people are in the audience I'm sure have experience what other challenges I would like to ask you of doing analytics research from planning to looking at this information to interpreting it what other challenges have you experienced and I'll pose that question to the online audience as well there's I don't think there's a mic in the back I think these are the only tiers so I was in a session like this last year and d2l was looking at similar data and there was a big issue of quality versus quantity sure they discovered that students who spent more time on a page and highlighted more did poorly more poorly on an exam and it turned out that there was an inverse correlation because the students didn't have any ability to discern what was important so they were highlighting everything and it turned out to be counterintuitive because we were looking for more activity when we should have been looking for more quality so did you see anything like that in your studies we saw that highlighting was the most common tool to be used we saw that people highlighted extremely large passages so large that it's hard for us to discern what might have been important in a passage because you can just click and drag so I can't really speak to the exact outcome but we did see a lot of very frequent highlighting as if to say that it wasn't sure what was important other challenges similar comment and I think and the whole notion of highlighting or using the note-taking tools in the books is really interesting that there's been some recent studies online the textbook's notwithstanding that highlighting actually doesn't help you understand the material that you're reading so whether you're doing it in an e-book or you're doing it in a paper book and I was a notorious highlighter in my textbook but apparently that wasn't actually helping me other things I did help me more so I think the real challenge in the analytics and understanding the science of the data is what actually is meaningful to look at we made some assumptions early on in our own e-textbooks that highlighting we called it was a proxy for engagement right so we didn't know what to look at so we started looking at highlighting because we could and note-taking because it was easy to capture and we were wrong so what is really changing student behavior what is actually improving student performance I don't think there are answers yet in terms of how the textbook and how content actually fits in there other than some of the things we know about reinforcement of content with short chunks of data and and quizzing often throughout the book but but the whole notion we have way more data than we have an understanding of what it means and so that's been my experience in my challenge thank you I think another challenge and I'm not sure if you guys had this experience with the vendor is that the vendors with the different platforms that they provide or tools provides as you say like potentially very rich data but also because they're not part of your research team the data is sometimes very complex or doesn't come in a way that is easy to analysis so I think for for Duke University I know our program evaluators talked about clickstream data and how it would be really nice to get into that but we'll probably have if we're going to get into it if we're it's going to require way more programming ability and capacity than than our small team can do on its own thank you yeah I think it's a very important interesting point it'd be interesting people's thoughts about that the time investment that goes into understanding and making it manageable hi I'm I'm new to the to the topic of analytics and my background is former English education professor and work now with helping out with the assessment of student learning outcomes and a little bit of quick context for the challenge that I'm wrestling with right now that maybe somebody can help me with I approach assessment with faculty as this is about asking and answering questions about how effective the curriculum is and so what you want to do is to go out and sample the things that the students are making or doing because they're participating in your curriculum and then you use a construct called a rubric to assess that performance and from that you can make inferences about how effective that curriculum is but you're looking at direct student products presentations papers examinations fieldwork as student teachers and things like that what's interesting to me what I'm hearing for two days about analytics is we're not looking at the actual student production we're looking at proxies I think that's a useful word I've heard you several times proxies for their engagement with the educational experience I find that absolutely fascinating I've heard you your presentation query the validity of what inferences you could draw about their engagement as readers through this study I don't know if that's a fair summary of what you what you're finding here question I have and I'm going to keep pushing is is there a way to wed what analytics is able to describe with the work that we're doing in student learning outcomes to go out and capture samples of authentic student work so that then we can come back and say something about the effectiveness of the curriculum and I'll end my long comment there thank you thank you that was a very prescient question and comment does anybody have a response for that I thought that there was a very interesting questions being posed in terms of collecting those other more relevant artifacts he said did he he said he asked if he got it right in his account of analytics I'm kind of testing my schema that I'm building here about analytics with you that's my old reading teacher background you build a schema and then sometimes you have to have that schema deconstructed in order to move on and develop a new one yes I see this as a conundrum between inputs and outputs you're measuring all the inputs how many pages they read how much they highlighted but you're looking for the outputs how did they do on the assessments or what's the quality of yeah so that's the bridge there and I think that's the holy grail for us if we can connect the inputs and the outputs then we have actionable analytics I think that's a very very interesting idea to consider we have time for perhaps one more comment and then I just have a couple more slides and there's and there'll be time for questions so it's not that I'm going to I just wanted to jump in and say that that we can't there's so many things that we can't measure and the things that we can measure aren't always telling us the things that we think they are so time on page for example I can read for an hour and I can look at a certain number of pages but you don't know really what I was doing during that time and so we make the assumptions that you know a student is actually reading it or a student because they clicked on a video and it and it played through that they actually watched the video that they didn't get up and walk away and so this is a problem we have with a lot of different educational technologies that the student does off-site somewhere that just because they've clicked doesn't mean that they were actually sitting in front of the device while the device is playing or showing the content. I think that that's just one of the most important things and sort of people have kind of commented on also looking at these inputs and outputs from my perspective an educational technology office vendors tout these tools as boosts to student learning and if we don't see adoption why have the tools and so I think that the agendas that people have kind of outlined are very excellent ones in our office you know we there's an assertion made about a tool and then there's myself and my team that's going to say does that does the assumption bear out and you're exactly right in the sense of analytics may show us information but it doesn't show us anything about the richness of that information so it may count some type of reading but it doesn't show us the quality of that reading and it is really important to to understand those limitations just just briefly going to just briefly going to wrap up here and just say that just because I want to have time for more questions is just that in terms of research if it's important to you to look at how people may be engaging in these systems and whether these are profitable things to look into I'm not profitable in the monetary sense but good for our students is that talking to vendors early was important so what we were able to do is negotiate a data dump early in the semester so that we could actually see what we were getting and actually make sense of it so that when the final thing came we actually knew what was to do I'll have a meeting with your IRB I don't think anybody for my IRB is watching me so I'm going to say this your IRB does not understand what analytics is my IRB is more concerned about how I recruit students and I could put all the information that I wanted to collect and I never received one question about that which I actually think is kind of concerning for the IRB as well but it's important to talk to your IRB if you're planning to do human subjects research which which we did and we received approval for everything we got and one brief example I want to give you about how this worked was that we got for example approval to collect all the notes that students put in the vendor did not want to provide us notes that were hidden from other students but we made the assertion that actually they gave us consent to give us all of their notes and so the vendor provided us with the notes there are lots of small negotiations like that that happen it's not as if there's an agreement that says we're going to provide you the data that you need in order to assess us effectively it's an ongoing it's an ongoing conversation and luckily we were able to have those conversations asking questions and more questions what does something mean there's a record here something happened what does that actually mean understanding and I think we've talked about it important to understand the the limitations of the of looking at big data and what people have talked about the cost of looking at it in terms of hours spent examining something like I said you know all of that information that we looked at the markup usage but because of the small amount it doesn't actually help us very much and consider where these subjective assessment you know in the form of reading journals really gave us a lot of good information that we needed and was accessible well designed and grounded in the literature about what we know about adoption was was was very useful for us just briefly what my team and I are doing now e-textbooks are not going away I've accepted that and they're very heavily adopted on introductory STEM courses so what we're doing now is more intervention work learning about the obstacles that students had to reading electronic content and accessing e-textbooks we're actually designing intervention materials that we're testing in these classes to see if students are more likely to to use them because we still tend to find that for example in our current study in an intro biology class that 80% of the students though required to have the electronics still by the paper so we do think it's important we do think it's important to look at this adoption I have references to this slide here I just want to acknowledge we received a grant from consortium of college and university media centers I took a picture from Flickr and I would take I believe we have about just seven minutes for questions so if there's other questions I'll take them now or comments including from online please you approach the just a disclosure I'm probably one of those evil vendors but one thing I thought was really glad you highlighted is we have the converse issue where we often want to work with people on developing these things and analytics is a particularly hard area to develop and getting the kind of support you have given your vendor in terms of getting involved early and having those constructs to define what we're going to collect because one of the biggest challenges with analytics is until you started seeing people using it and have those discussions often you don't know what they want to collect and framing that is very very difficult so the first thing my developer will ask me is what do you want me to collect what data do you want me to collect and it's just hard to come up I'll just click whatever you can collect everything every click every every little second so that's a big challenge and the second thing with analytics is at least a lot of the discussions I've been having with a lot of people I've been involved in this only for the last about six six seven months and every time I have an analytics discussion analytics is seen as the solution it's like it's it's not the solution it's it's a tool to support something else you're trying to do and it's important for us to understand what is that thing you're trying to achieve that analysts try to support it's I have a bit of a marketing background you look at google analytics and years ago we figured out time spend needs nothing you know that's why google has spent so much time setting up things like conversion and goals and stuff like that and that suddenly makes sense for marketing so I think the similar thing is apply is here and analytics is just such an early stage that until we understand what we're applying this to and how this is going to support pedagogy I don't think we're going to be able to solve this conundrum that we've got with time spent so it's just a comment thank you I have an online question do you do have an online question any correlation either positive or negative between time spent reading and outcomes example grades on assignment overall class grades yes so this is there is no correlation with time spent reading that we found what we did find though was in the data was an interaction between time spent reading and frequency of bookmark usage which could and I'm hesitant to say it because we haven't replicated it but in this study we did find and it may be explained that frequent reading along with a bookmarking strategy that enabled people to find information may have helped them to be learning it to learn better and it does make sense because the bookmarking tool was helpful for finding things but I would hesitate to say that we found that once and we haven't tried we haven't replicated it it does the system permit one student to read another student's e-text it permits a student to read they're they're sharing the e-text it permits them to read their notes other students notes okay but if they're shared could a friend lend a student his access to the e-text book not in this case they would they would share their university password the text book was contained in the lms well I hesitate to say that would never happen but I was thinking it might provide an explanation for the people who claim to have done reading and didn't show so in the system and might also present an additional complication and analyzing this e-text book and which might actually be why it was hard to access those people is that it was buried in the lms so so so went the lms so went your e-text book we did a similar e-text book study at the University of Kentucky a few years ago and it was spearheaded by our central it academic technology team it sounds like you had a very rigorous research study approach and I was wondering if you could talk a little about who was on the team and you know was it a partnership between it and and college and faculty and so how did you create that partnership so that's a good question uh my one of my mentors so professor cap issue uh was a mentor of mine when I so my PhD is an education from Iowa um because she has an educational psychology background we recruited her to do this study with us to help us design it um and Jane Russell is also has a PhD in educational psychology so um so our training being an education helped but we also had to seek out some people talked about teaching elementary reading or someone talked about that briefly and what we found we actually looked around and we couldn't find like a good measure that applies to adults for reading behavior I went to our education library and there was tons of information about kids there's all this there are all these scales there are all these little tests but we finally tracked down this reading behavior questionnaire that was published in the literature um that we found to use so so we we were lucky and that because we had the grant the grant required us to design a study and have external people vet the study as well before as a condition of getting the money so that uh somewhat behaviorist incentive helped us a lot because it actually helped us think things through and and and we and a lot of our research has done this way we have education people work with us because they bring a very good perspective um they um they're not technology people so they don't have the same biases that exist in my group which is very helpful um to checking those biases and ensuring that you're looking at the at the data fairly and not just being happy with shiny shiny things so I believe I'm out of time but I will be up here if you have any other questions or comments and if you'd like to approach I'll thank you for your participation and for your willingness to read about biology the two of you and uh and thank you for very much in the in the middle of the dais I'm not sure if people thank you but thank you for all your help as well thank you thanks for coming