 I've been using hypothesis in my classrooms for five years now, and I'm here, I'm going to be, I have a couple of things I'd say for, first of all, I'm Juan Pablo Alperin, I'm an assistant professor at Simon Fraser University in Vancouver, Canada. I teach in a publishing studies program, so and I do my research and my work on scholarly communications, which is how I first attended, I annotate back in 2013, thinking about it from the publisher's perspective and the world that I do building open source software for managing and publishing academic journals, but I was immediately attracted to it as a tool for teaching and learning. At that time I was a PhD student and I started using sort of annotation in my own classrooms, while before hypothesis was really ready for primetime, and I've then, since I've become a professor, started from my very first year trying this out into my classrooms. I have a tendency to speak quickly, so I'm going to continue on that tendency because I only have, I forgot to start my timer, so because I have a lot to get through, I'm also going to rapid fire through a lot of graphs and statistics at you from the research that I've been doing around annotation, so you might want to look up if you're interested, because the slides and the images are going to be coming and going at a rapid pace so that I can get to this wealth of data that I've collected over my last, during this last semester and the work that I've been doing. As someone who is sort of very passionate about teaching, I naturally found myself asking for two course releases from my university to be able to do research and to promote the use of online annotations, so over the last year I've spent time at Santa Fraser University running workshops and teaching faculty and trying to encourage this practice, and over the last semester I had six classrooms running, hypothesis in classrooms other than as well as on my own classroom, and so then I was, I've managed to collect some data and interview and survey the students and that's what I'm going to talk to you a little bit about today, trying to learn a little bit more about how, why, and what students actually annotate in different courses. I just want to give a little shout out, and often this comes at the end, but I think it's important to put out the beginning that this work wasn't, isn't work that I've done on my own. I have two research assistants that I've been working in my, in the Scholarly Communications Lab, which is my research group, Alice Relackers and Esteban Morales, and Remy Kallir who you've all heard. I met him at I annotate two years ago and we decided that we should collaborate on something and so I brought him on board as sort of being the person that's actually doing research and education. This is really not my research area, I just something that I wanted to do, so I thought I should bring someone from education on board, and Remy and I have been working nicely together on this project. I'll just start off by sort of a little bit, this is from one of my earlier, I presented this at I annotate two years ago. This is probably my favorite quote out of all the surveys and times that I've had feedback from students. I just want to say like a student that gives at the very end of the term saying, I enjoyed reading the annotations alongside the text primarily because it helps me engage with the text, the sentence level. The product of an educational system where annotation and critical reading was not encouraged and not taught, so the hypothesis tool really helps me understand how to read critically as opposed to just absorbing information. I've actually asked my friends, students and professors outside of Vancouver to use this tool. This sort of like captures like why it is that as a teacher, as someone that's trying to encourage learning, encourage critical thinking skills, hypothesis sort of seems to work for some students. My second probably favorite quote is the very honest student who says to be completely honest I was never one to read could read every reading fully and this forced me to and they recognized that this is not a bad thing. So I just want to say this is not just about all of the oh let's just think about all of these great pedagogical outcomes. This is actually a great pedagogical outcome from maybe a little bit of a different perspective but it's really is really true that hypothesis does different things for different students and I think I wanted to sort of dig into this a little bit a little bit more. So I was saying I managed to get this teaching fellowship at SFU where I would promote annotations across the university, several courses. I ran, I was running it in six classrooms. I would consider three of those running it into those out of those six classrooms I would consider three of those to be have been a successful implementation of hypothesis and I won't really have time to dig into some of the reasons why the other three weren't successful but something for us to if you'd like to sort of catch me and we can talk a little bit about at the break some of the barriers that those other instructors found and the reasons that those courses didn't have enough adoption for me to sort of say okay this is a course actually worth for us to dig into and study more what happened because I actually just think it becomes a question of it was a poor implementation not poor outcomes of the use of the tool. I collected all of the annotations from the courses through the API we surveyed all of the students and actually got and one of the courses is only eight students we got all of the students to respond and the other courses we had over 50% of the students in the class respond and then I interviewed each of the instructors in the courses so that's why I have a lot of things that I need to run through and dig into because we sort of collected a wealth of information about three courses that are in themselves not so big in terms of the number of students but sort of we have a very full picture of what went on into those into those classrooms just again overview just some numbers I told you you're just going to be a very sort of numbers and graph heavy presentation I just want to give you a sense of the scale that we're working at if you look at the number of users in each course this is the number of annotators in each group so in one of them is so it's 10 there was eight students and there was a TA and an instructor making 10 the other courses around just a little over 20 around 25 students annotating in each course and the volume of annotations varied quite a bit so in one of the courses the other publishing course is a colleague of mine in publishing she's run she's used hypothesis before in her classroom and so and somehow the students in that class sort of annotated you know almost sort of over four times as much as in one of the other courses even though that had the same number of students and annotated sort of somewhere a little bit more than the course that had 10 10 students so again you see quite a range in terms of the number and volume so when I show you some of these graphs where we survey the students and we look at the annotations keep in mind this is coming from a different number of students in each one so this is just sort of when you for you to refer back to and have in your mind around the the different things that happen or the different kinds of classes just to give you just over your it's a Gero is a gerontology course the GSWS is gender and women's studies course and pub is a publishing course so very different subject matters and all three of them give you a different sort of sense that the kind of annotations that students do actually varies quite a bit by courses is just to give you a sense of them the sense of how long the annotations were and so you see that in the pub 480 course and there was more annotations but also the annotations were sort of on average longer than the annotations in the other courses and you get always a bit of a range right so it's like why as soon as you're getting into those dots that's the 25th percentile that's sort of above the sort of beyond the the one standard deviation from the fifth percentile there where you're seeing like some students are really annotating really long things right and then some students the core of them are sort of around 250 characters and for the other two courses probably I can't quite read exactly it was probably around 150 characters or so so about the length of a tweet for the median annotation or the old tweets not the not the new tweets I've been on twitter for a long time. Another kind of differences so we're just looking at some of the different things number of URLs included in the annotation so trying to get a sense of how much are they linking out huge range so in terms of out of remember there was over a thousand annotations in the pub 480 course there's about four URLs in those annotations in the gerontology course it was around 750 annotations and there was 34 URLs in those so not that many things linking out but again quite a range between the different courses and this is sort of give you a sense of the different annotator types and this is actually something that in all the classes I've looked at so beyond even just these three on the one side of the graph you're getting on the left is the number of annotations per person and and they're just like in a rank order so they're the the place where they are it's like the further up there the person that annotated the most on the other side is the median length of that person's annotation so what you see is that some people annotate a lot but very short annotations so those are all the downward sloping lines some people don't annotate very much but annotate very long and then some people are sort of more in the middle and right and so you get sort of the different kinds of annotators and this is just to show you one course in the course with the smaller students you still get that same kind of like crisscross effect where you just get a sense that there's different types we haven't done very much around trying to classify annotators into this group which is sort of then this is the preliminary stuff to give you that sense of the different kinds of prequel that we find even just based on these two dimensions of how frequent and and how long then we surveyed the students first we wanted to get a sense of first their motivations for annotating and they're coming in so we asked them questions around like beyond this class how often they annotated just to get a sense like are you generally an annotator and this is sort of giving us that baseline parameter of are these students coming in students that already have the practice of annotating regardless of what form what you find is actually a pretty sort you know you get like in two of the classes about 50 percent of the students were on the agree side very few students sort of like are strong annotators saying that they strongly agree and one of the other courses students are really very much not annotators actually it's in the course where they did the most annotations in the end are students that were very much on the disagree on that side so just getting your sense of that baseline when we asked them if they liked reading a signed text again this was just like you like reading a signed text in general you know the students are again you know some in the in one of the classes the students that there was you know 30 almost 40 percent that strongly agree most of the students are a little bit more in that middle they like it they don't love it it's just part of what they have to do as students just giving you that sort of baseline we asked them to stop and questions around if you annotate it can you tell us why you annotate it that we haven't done this is just again very preliminary we've only just the term just ended a month ago so we've just started going through them we've tried to pick out the themes that we're seeing without having fully coded them all yet but just to sort of putting the these are sort of in rank order that we're getting a sense of from a first pass okay the main thing that comes out why they're motivated is because it's course requirements and marks in fact one of the courses where it wasn't successful the annotations were and made mandatory and so that was one of the reasons why perhaps we didn't see so much but personal learning and peer conversations also came up so those themes around motivations for learning and community building are themes that come up in open-ended responses around why they annotate it when we asked them why they didn't annotate first of all and this is the one sort of I highlighted some that jumped out of me as interesting around like no they say they had no valuable ideas to contribute and so that's something around they need to like they're not they are not feeling like they have something to add to the conversation and then tech issues and having issues around the technology was sort of a second theme that came up a lot and the third is students that are really not finding the value they're saying I don't don't see why I would be doing this and again all of this points to there's pedagogical strategies around how to encourage and motivate it in different ways to make this them see why it would be useful but this is what they report then we got into some different themes around community building this is where if you just sort of want to take a look without looking without necessarily needing to worry about all the numbers you just kind of look where the graph skews right that's sort of if you want like they're not having to think very hard right now because it's the afternoon and you have an attic break just sort of look is the graph mostly on the agree side or mostly on the disagree side they feel that annotations is helping them to share their knowledge with their peers so there's something like this is like they're very much in the strongly agree and agree camps right so they're very much on that right hand side sorry left hand side of the graph on that agree so there's a community building component that really seems to be something that the students are recognizing when you're asking them around whether help them understand other points of view and the potential for annotations to sort of help build consensus and help bring other people's perspectives again we're very much on that left hand side of the graph so students are recognizing a value around that these community building or building cohesion or building shared understandings of questions when we asked them around sort of different things we asked so this is just a few of the questions we asked around perceptions of learning whether they felt like they were learning or getting some greater understanding or building knowledge asking them if they made them think about course content and concepts beyond the classroom again we see that the responses are have the students sort of well over half the class in some cases almost 80 percent of the class is very much in the agreeing that the annotations help them to think beyond the classroom this is just self-reported perceptions of of what the annotations did for them and if you ask them this was sort of this is like that survey that you get asked when you participate in anything and they say would you recommend a friend to would you recommend someone to use this this is sort of our summative question that gets us that sense of like do you think it was actually useful for learning just asking them overall our annotations using this tool did it help you learn and again we see that almost nothing on the disagree or strongly disagree side so everyone is either from neutral with a small percentage and then the vast majority are actually on the on the usefulness side so this is students are reporting that this is useful so those of us that have been using the tools in our classroom for a while everyone that was sitting up here minutes ago that I know this is going to come as a surprise but we wanted to sort of get a sense of documenting it I want to just touch on a last sort of bit of giving you one more piece this is all student reported things around what they think what we try to do is that we actually use sort of a framework proposed by Pliwinski and others looking for evidence of knowledge construction activity so it's trying to think around like what was actually happening in the annotations and so Pliwinski sort of outlines different activities different things that we might classify as knowledge construction so that whether they were using the annotations for clarification that whether there was conflict and disagreement these are all places where learning happens or knowledge is created right building consensus elaborating on an idea interpreting something even asking a question is in itself is creating knowledge by even just by the sheer act of questioning and triggering there's some thinking that has happened and even the understanding that you have a question there is in itself useful or support and like and empathizing all of these different so we quoted for all of these different kinds of activities and we took the annotations from the all of the courses and tried to see which of these things are present so every annotation could have one two more or any number or even zero of those different examples what we find is that interpretation and elaboration happen the most with question asking coming in third so this is giving us a sense of if we if i get asked did learning happen in the annotations this is what i would sort of point to like these are the kinds of learning and knowledge construction that took place on the annotations that wouldn't have happened if the student was just reading on their own or it might have just happened for that individual student but it didn't happen in a collective space for all of the students to to take advantage of so we're starting to get this sense and we're digging into these now a little bit more and trying to see get a better sense of what these what these things exactly look like but these we've actually coded all of the annotations for the for these three courses and so we sort of asked them and just to finish up sort of quickly we wanted to sort of know okay we've seen you're reporting that you're learning you're reporting we're looking at the annotations and we're seeing evidence of certain knowledge construction of certain types of learning we also asked them what a think of an annotation that's sort of useful to you and tell us why did you think that that annotation was useful so pick of an example that's something that an annotation or example that made you that makes you think that annotation is a valuable practice and we got the students to report one is that it caused them to think differently so this is very much I think what people imagine of as like happening when someone learns it's like oh there's an idea I hadn't thought of before right but interestingly also students report usefulness in reconfirming their worldview and this is something that we probably see online and if we were to get into a discussion around discussions commenting and and anything that sort of happens on the web where there's very much around the social practice that we look for ways of confirming our own identity and confirming our own beliefs and this is something that students are actually actively reporting that annotations is helping them do it's helping them to understand difficult concepts reading and also just realizing and they find value in having helped somebody else like when they make that even when you ask them what was useful about the annotations instead of reporting someone else annotations helped me they found it useful when their annotation helped somebody else I think it just helped them to reaffirm that they had ideas that were worth sharing which is something that we very much at least from my philosophy of trying to promote for public knowledge and open knowledge around sort of helping students learn the value of their ideas and the putting them out there is worthwhile for me is a sort of a very rewarding thing to see reported by by the students I want to just finish off with a couple of last quotes from the students just to think when we ask them how would you like to think them use in the future you know students say I think annotations should be incorporated into every class with weekly course readings it ensures that students are thoroughly reading the course material and coming to class with a sort of right from the surveys of these students that we're just talking about students sort of there is like a sense of like they want this they want to see this everywhere because they realize the whole class is coming more prepared to participate and just to finish it with sort of saying that not the experience of loving this is not universal I don't want to be here just by advertising and promoting all of the virtues all these things but I love it just I want to say I don't think it works for the future because many people wrote useless things and so this is not everybody's experience is the same and I think that's important for us to finish off so I'll just sort of finish with these sort of lessons that I think take us what we can learn from the classroom that make us think about how we might use annotations or in the in the broader context of thinking them out there I think that annotations we have evidence that where it's a place for learning happens and it's a place where other learns what other are thinking and where they want to share their thoughts with others I think that's like the evidence that I've collected from this study sort of tells us this but not everyone is motivated to annotate when they start right so there's something about that intrinsic motivation of the grade that's helping all of this stuff happen and not that nor does everyone find the same value in doing them right people are reporting different things is valuable and some people are reporting that it was not valuable and so that's something for us to also take into any context beyond the educational context and then the last thing is that how it gets presented and this is the part where I didn't really get into the discussion around where it didn't work into some of the other classes how the how you present it how the tool actually works around removing those that friction and who the community members are that are participating in the conversations really matter and so there's a lot of sort of nuance and detail and things that we could unpack from that sort of sentence that sort of captures saying different context and different people make the experience different which is obviously true but it's something that I think we can point to a lot of different specific examples from these classrooms where we can start to sort of pull that apart and try to figure out how if you're wanting to use annotations in different contexts you might some lessons to be learned there on how you might do it in a way that leads to better outcomes. So how are we gonna encourage annotations in general that's like a question that I think I leave you with and more importantly maybe how are we going to encourage sort of high quality annotations that lead to the positive outcomes out of the things that we see that we agree that we want how are we going to encourage annotations to take those formants for those communities to form. Thank you. So if there are questions for one I'd like if people would come up to the microphone it makes it easier to capture on video and make sure everyone can hear what you're saying. Thank you that was a great presentation really considering how we do some surveys too did you think of asking them also whether or not they would recommend this tool for use in other places where it's a classroom or to their friends as a possible way to get back how well they enjoyed annotating beyond having it to be for a grade. I've done that in previous classes that because I've surveyed my own so I've been running it in my own classrooms for five years and I've surveyed my own students and I've asked them a question very similar to that in this survey the question that we asked is how would you like to see used in other classrooms. So we say specifically to the classroom setting and then asked them beyond that a few of them because there were some open-ended questions like for that one for example so we get some responses that come up with things that point to beyond the classroom examples but for the most part they sort of talked about different ways of integrating it saying that they wanted it done the same way as it was this time but for all the readings or the sort of pointing to wanting to see it more examples around or getting clear rubrics for how it was great so they started pointing to things within the classroom context but not so much beyond it. And one other thing remember did with some of the ones where you said maybe they did not implement as well do you ever find that maybe annotation doesn't work around certain contexts or is it really just how it's being implemented. In the classrooms in my in this case where the other three classrooms where the people that have agreed to the instructors I had had conversations and they wanted to implement it and it didn't work it I don't think in any of those cases it had to do with the content there was one classroom where someone wanted to do it but they were using a textbook an online textbook and the textbook technology they wouldn't wouldn't work with hypothesis so that was one where it was very much the the version of the book. In the other courses it was really more about some technical problems the motivations one of the classrooms I hadn't realized I was just having this conversation a little bit earlier I hadn't realized how important was the way that I've learned how to motivate and encourage us to use in the classroom has turned out to be really important I saw a classroom where there was almost no uptake of annotations and so I was surprised because it seemed like the kind of course content and the way that it was set up should work I saw I said can I come into your class and talk about it so I spent 10 15 minutes at the beginning of one class about two-thirds of the way in and for the last three weeks the annotations jumped a significant amount so there's something around like I could I had the experience and I could tell them why they would find it useful and I could really explain that in a succinct way and encourage and then that worked to get them to turn it around so it was really more there is a lot of care and craft that needs to go into putting making this work in a classroom and I think that even these instructors will do better the next time around in terms of the kind of uptake that they get because they will all they'll know the benefits themselves they'll be able to explain it to the students one I actually had a question yeah I noticed in the survey data and this maybe you know I think the end was 12 or something for the publishing class which I think had the most students in it yeah but I noticed they were in the survey data responses they were the ones who had the highest negativity toward doing it again I think um they can yeah I'm not I mean we can go back I'm not sure which are the depending on whether whether they would uh whether to help them learn there um so you see pub 480 disagrees oh yeah it was yeah so this is more around like whether to help them to think about the course content which yeah so we actually for each of these we have like five different questions around motivations five different questions around community and five different questions of our perceptions of learning and I've only presented two from each one here just in the interest of I thought that was probably fast enough for everyone uh but uh yeah we but I think the students in that class I wouldn't characterize them as being negative when I look at the overall responses from them um like I said that colleague is one that has run this kind of has worked with annotations before and I think that she's like the students she keeps doing it because the students do have a positive response to it questions for one