 whether that was to solve a problem, design a solution, provide some perspective on a project. And I know that you do that not just for me, but you do that for a whole range of educators in both K-12 and higher education settings. Now, I know that Hypothesis isn't expressly an ed tech tool, nor is it an educational expressly organization. And yet Hypothesis in my assessment is one of the most responsive discerning and inspired organizations that's impacting educational technology. And today we're going to hear about why that is. It's my time to get out of the way. I really want to focus on our esteemed panel this afternoon. And we'll be joined by four accomplished educators who will discuss their experiences as designers, as facilitators, and as researchers of learning. Now, contrary to the popular trope of the overwhelmed or disempowered educator, our panelists today will highlight a counter narrative of creativity and agency whereby educators who care deeply about their students and their students' well-being and their students' learning are eager to experiment, to iterate, and to inquire so as to create more engaging, more relevant, and lasting learning experiences. And of course they're going to do so by talking about annotation. So today we'll hear from Juan Pablo Alperen, Larry Hanley, Kelly Hutchinson, and Brian Luckoff about how educators are again designing, facilitating, and also measuring learning. And whether they're in classrooms, or they're online, or in some hybrid in-between space, we'll hear about the power of annotation and how that's impacting why their students are learning. Practically speaking, our presenters will share for about ten minutes, and then we'll follow that with a few minutes of question and answer. And I will briefly introduce each of the panelists before they present so you have a bit of context and so they can set up and move through their slides and information. I'd also like to add briefly that some of their slides and some of their information is already loaded into our shared notepad, and this is of course accessible via the schedule. So we're going to dive right in. And we'll begin, if I'm not mistaken, we're here with Juan Pablo Alperen. So Juan is an assistant professor in the publishing program at Simon Fraser University, as well as the assistant director of research for the Public Knowledge Project. In both of these roles, Juan is an open access advocate and a researcher of scholarly communication with a focus on understanding the value that the public finds in scholarly work. He believes that by having students work in the open, they will understand the value of public scholarship and become more publicly engaged citizens. He's been studying his students' use of hypothesis in the classroom and is interested in figuring out ways of encouraging students to remain engaged and to improve their learning outcomes. Please join me in welcoming Juan. Alright, so thank you for having me. Thank you for that very generous introduction of this whole panel. I set up some very lofty expectations that now I will hope to try to meet. We'll see how well I can do that. So what I want to talk a little bit about is I've been an assistant professor now. This is third year now that I've finished. So I'm fairly still new at this game. But I've been experimenting with doing online annotations in my classrooms for these last three years. I teach both, I teach in a publishing program, both upper level undergrad and a seminar course that both have, and a project course in which I don't use hypothesis, but two courses a year where I've used hypothesis for the last three years. So I want to share a little bit some of the insights and some of my motivations for doing so as well as some of the things that I've found. This very last iteration when I ran the graduate seminar course, I got some, a little bit of funding from the Teaching and Learning Center at my university to be able to do a little bit of research into how the students were annotating. And this is something that I'm hoping to now replicate as I go into next year as part of a fellowship that I've received from the university to try to study annotations across many more classrooms. So what I'll talk about today will just be about this, talking broadly about my motivations and my reasons for doing things as well as the experience. But then I'll talk about the experience of just this one classroom where I ran it and the surveys and the analytics that I ran on the students' annotation as a way of getting some insights into what is going on when students are asked to annotate. First I'll just talk a little bit about my own motivations because, and of course I think for other people that are working with hypothesis in the classrooms and I'm sure even from our panelists here, there are reasons for wanting to do annotations that might be very different. But I think it's important for me to make clear what are my motivations and what are the things that I'm trying to accomplish with annotations because I don't think that there's a single model or a single way of doing annotating in the classroom that works and the way that I've done it in the classroom is certainly based on and inspired by what is the reasons why I think students should be annotating in the first place. And like was sort of just said in the introduction but for me is that I consider myself to be an open access advocate, so open access to research. And I see this as my way of doing annotations as a way of trying to promote open access to knowledge. And I see it by having the students be seeing the value of putting their work out into the public sphere and having them understand that value of openness through their own work as well as their ability to annotate things that are openly available and seeing what the value of that is. Asserting, the second thing the public mission of university is something that I feel is very much under attack and I think that this goes in both directions. Again, it's about the students showing the world that they have things to contribute and showing the world that things that are going on at university have some public relevance and annotating things publicly is a way of doing that. And from the outside, for the outside world to then see that value and to all of a sudden realize that universities have a public mission and they're somehow fulfilling that. That's a little bit even more lofty than the other two if those aren't lofty enough is fostering some civic engagement and getting these students to try to want to engage beyond the classroom afterwards by feeling comfortable in making public contributions. I think it's something that annotations have the potential to do and something that inspires me in trying to do my work. I try to teach students to be open in many different ways by making all and doing... Annotation is just one of the many open practices that I do in the classroom. I make all my readings open access. I have students publish all of their work. I give them all of the feedback on their essays and through annotations and as well as I have students peer review each other through commenting. And sometimes when I teach on the data analytics I also have them use open data sources. So again, it's all part of open practices. And I see many pedagogical advantages to doing and some of these there's been some research that's been done that's part of what I want to do in the next year is to study a little bit more. I see annotations as an avenue for students that are more shy that don't feel comfortable speaking out in class to have an opportunity to participate in that classroom setting. I think annotations give students a window into how other students read. It becomes a layer of exposing what other students are thinking as they're going through the text as well as myself. I think it causes students to actually read closely without skimming all the way through the very end of the text because they know that I can see how far they've annotated. And they know that their peers can see how far they've annotated and so they know they have to keep reading because they're going to have to be engaging with that text and there's some layer of transparency as to how far they got. And it exposes to me what the students found interesting and allowed me to adapt what I do in my own classrooms. The night before I go into the class I go and I scan through and look at all of the latest annotations of what they've done and I tailor the discussions in the classrooms to either address what they've already discussed or to touch on things that they have somehow missed that they didn't look at. And so making that visible for me has been something that I think has improved my own teaching and there's this one I conducted a survey of students both at the midterm and at the end of the semester and there was one quote of the students when there was a part that just said what are the anything else that you would like to add and I think this one student really captured the spirit of all of the things and all of the reasons why I think annotations are important and I just want to read this out. It said, I enjoyed reading the annotations alongside the text primarily because it helps me engage with the text at the sentence level. I am the product of an educational system where annotation and critical reading was not encouraged and not taught and so using the hypothesis tool really helps me understand how to read critically as opposed to just absorbing information. And then she went on to say how she's been promoting and encouraging other people to also use hypothesis. And so what I decided to do was to take these ideas these my motivations, I've been doing it for three years but I hadn't tried to really measure the effect to see how these annotations were working in the classroom. And so what I did was build a little tool with the help of a developer to capture all those annotations through the API and then to just produce some very simple metrics and I just want to just run through them very quickly as a way of giving you an understanding of how students, they don't all annotate the same and they don't all engage in the same way but there's some interesting things that I already was able to find even just this is just again from one course don't take these to be definitive of all annotations. First is I found that not all students annotate the same amount so you get some super prolific annotators. This is a total number of words annotated over the course of the semester and then some students on the very right DM and HD annotated a lot less throughout the semester the number of words that they contributed to the discussions was a lot less. Interesting, one of the students sort of in the middle towards the side are the ever spoken class. Someone that was actually not very comfortable with her level of English but she is sort of the fourth or fifth here on this list of the most vocal in the online space. So those are, you start to see some of those dynamics play out. Annotations from week to week really show when students get busy and when they're not so there's a couple of weeks, three and six there was no class so you see that there's some variance there. Seven is when I gave them their midterm grades for the semester and so you see that there's a big spike coming right back up on week eight after they realize the ones that we're not doing as well decided they should once again re re-engage with online annotation. So it's variable and as the semester gets busy they drop off at the end as their final term projects come due, right? So students, their ability to engage critically varies based on how much throughput they also have. It takes time and effort to annotate so they are overtaxed with other courses their ability to do so diminishes. Curiously, the students who annotate more don't, it's not that they write a lot of short annotations. Annotations had about 40 words on average per annotation and regardless of how here we have on the on the x-axis is the number of annotations right with a student going over 300 annotations and on the y-axis we got the number of words per annotation on average for that student and we find this fairly sort of, you know, there isn't a big variance that if you're just annotating a lot of short. So 40 words per annotation was what around what these students were doing regardless of whether they were writing many or few annotations. And then, you know, this is just some of the metrics that I was able to calculate based on the annotations themselves, but then I also surveyed them to try to get a sense of do they think it's helping them to learn, right? I didn't actually just did in hearing that there was an NSF study to try to measure, for example, how effective these things are in helping to learn, but I thought it's important to at least get the student's sense of how much it helps them. So one thing is that, you know, asking them does the annotating enhance or learning students on a four to seven scale are slightly on the positive side, right? So they tend to, and this is the blue is the midterm and the green is the final same students just asked at two different time points. There's not much change, but you find they're just on the positive side. They're also not saying that this is, you know, a seven in terms of helping them to learn. Do they like doing it? So, you know, I asked them if they found the readings interesting because I felt like this would probably, you know, somehow affect if they're not finding the readings interesting. They're probably not going to find annotating them interesting. And so they're saying, you know, between five and six out of seven but then when I asked them do you find yourself annotating because mostly for marks or mostly for other reasons they're more on a slightly on the negative side, right? So you actually get quite a big spread. If you look at, you know, the outliers there and the quartiles, you see that it, you know, it varies from all the way from one to seven on the answers. But they tend to be slightly on the, with a median on the slightly on the negative side negative being doing it a little bit more for grades. So this is something that I think really requires a little bit more work in how to motivate it properly for them to want to do it. Since they said it helps them learn why are they not then annotating because of that motivation as opposed to just because I'm grading it for it. But if you asked them how much they enjoy using hypothesis then now we're back on to the positive side of the scale, right? And with a median of five, right? So that's so that the use of the hypothesis is enjoyable but still doing it mostly just because I was giving them an online participation grade based entirely on their annotations. And then just last bit of sort of statistics. I did a qualitative assessment of their annotations on a week to week basis on a scale of one to five. I graded them. Five being they made a very meaningful contribution. So this I was trying really hard not to base it on how many annotations they made, but rather on how good their contributions were. And what I found is that, you know, that's the bar graph on the left-hand side. You can see it's sort of a bimodal distribution where some students were sort of middling in the qualitative annotations where students were sort of consistently good from week to week and making really strong contributions. And so that's sort of an interesting distribution. All the students that were annotating were making decent contributions, right? So that's they're all sort of in the two and a half to three. But then some students were really four and a half to five on average per week. And then what I did is just correlated that to the number of annotations to say, could I save myself the trouble of trying to assess them in a qualitative way, which has actually required me sort of every on a week to week basis evaluating these. And I just correlated that with the number of annotations. And you do find that there's a positive correlation between the students that are good at annotating are doing more annotating. And it's just there's there's something about how much they're engaging with the text. And so there's something there around trying to figure out what are the ways of getting those students that are not annotating so much to also want to participate. So just to sum up a little bit, what are a couple of things that work well work really well. I think that most students do seem to engage well with the texts, right? I think the students were there week to week doing things and contributing and reading through all the way to the end and making some contributions. Students are reporting that the annotations are helping them to learn. So that's, again, something that I'm hoping to study in much more detail to see if we can actually measure that effectiveness in a concrete way. And it really helped me from the perspective of the person that was in charge of running the seminar on a week to week basis to see those annotations. So I think those are three of the core things that I really liked about this experience over the last three years. What I think still needs work is that I think I don't like the idea. It doesn't sit right with me that the idea is that they're annotating because I'm grading them for it. And I really want to experiment with different ways of explaining and presenting the role of annotations so that they want to annotate because they're finding value in it. I think I must not be doing something right if they're still reporting that they're just doing it as part of doing school as opposed to as part of what you wanted to do something. It's a big challenge in all kinds of things as instructors we find, but I want to get through that with annotations. And the second thing was just something that I was really trying to do with this project was to figure out how to get students to go back to the text multiple times. I started building a notification system to try to encourage students to go back to a text after they'd annotated it. So go back a week later, go back if they were the first one to annotate after there was five more students that came and annotated it they should be going back to read and engage in that discussion. When I measured that it turned out very few students did that and sending them a notification wasn't really working and getting them to do that. I want to figure out how to work on. And there's a bunch of other things that I want to try sort of experimenting in different ways. One is assessing various aspects of learning. I think asking the students if they're finding it helpful there's one thing but I think we actually need to try to measure it in some more concrete ways to see how helpful are annotations and helping students to understand the course context how helpful are annotations and getting them to be good critical readers and how helpful, how can we try to measure the quality of their annotations in a way that isn't or in a way to see if they get better at annotating over time because having being a good annotator seems like a valuable skill. As well as experimenting with additional features like other kinds of annotations as well as having a like but other things that might get the students to want to be annotating and to get better at annotating. Overall this is again just some stats from just one classroom. Next year I'm going to be running this across a whole bunch of classrooms at Simon Fraser University and the idea is to try to see if we can pull out some lessons that will be useful for all instructors that are trying to bring annotations to their classrooms. Thank you. Greetings comrades co-conspirators I bring you welcomes from the hardworking students of San Francisco State, the hardworking faculty and staff. This semester I'm teaching a course called the literature of labor and this past week we've been discussing called sleep dealer directed by Alex Rivera in 2008. Has anybody seen sleep dealer? No? Oh my goodness. It's an independent movie. It's been getting more buzz in the past couple of years. Sleep dealer is about a movie about a young man who lives in Oaxaca Mexico and for various reasons has to move to Tijuana and it's set in the not so distant future and he works in a virtual maquiladora, you know the factories on the border. It's a fascinating movie about digital labor, etc. And most importantly it's a really interesting treatment of the border which of course as we know is a very lively topic these days as well and the way in which borders both connect and separate the ways in which borders are potentially sites of violence the ways in which borders are figures and very material instances of exclusion but the thing I like about sleep dealer the movie and you're probably asking yourselves why the hell is this guy talking about some movie nobody's ever heard about or seen one of the reasons why I like the movie is because it's also about how borders are the places of incredible creativity hybridized and variegated identities and experiences and how borders can also foster new solidarities between those who have been divided and what not and how borders can without borders you can't really have for instance hacking as a cultural and perhaps even a technological trope Now why am I telling you about this because I think one way I like to think about digital annotations or web annotations is in terms of borders that is annotation online annotation I use hypothesis as a plugin in wordpress annotation creates a border between a text and another text or a text and another voice in this case in my case students voices or students writing we often think of annotation in terms of a kind of very hierarchical or you know often hierarchical kind of model where the annotation is subservient to the primary text or as an adjunct to the primary text but what I'd like to propose to you today is that we think about annotation as really creating this interface between two different texts and an interface that in and of itself can be an extraordinary and fascinating and productive site of creativity and knowledge and what not okay so I have four theses or four ways perhaps of thinking about annotation in these terms one is annotation makes visible in new and more social ways the ways in which meaning is a provisional collaboration between readers and texts rather than something to be discovered and getting deep within texts this is important right because students and readers generally think that there is the text and somewhere beneath the text is the meaning of the text and yet I think one thing that annotation can do or make visible at least is the process or meaning as a process meaning as a collaborative process constitutively not as you know here's a book and now we're going to read and discuss it but the story depends on this communicative and collaborative dialogical process so I'm just showing you an example here this is from one of my classes I think last semester or this semester rather where the students start engaging in this you can see Maureen over here on the right how many of you have read the yellow wallpaper what a great story it's like have you seen the movie the grudge grudge or crimson peak you've seen crimson peak the del toro movie okay how many of you have seen a god damn haunted house movie raise your hands if you have seen any haunted house movie thank you so then you must read charlie perkins gillman charlie perkins gillman's yellow wallpaper because it is like one of the greatest haunted house stories it's very short too by the way in american literature so here you can see maureen says john is obviously the antagonist in the yellow wallpaper this is a very important and kind of controversial way of reading the story and this then produces this dialog amongst the students about what does it mean to be the protagonist how does the story work so in other words the meaning of the story is not there in the text it's in this process of the students collaborating talking together and working together alright I'm going to give you one more idea I'm going to try and give you two more if I don't run out of time okay online annotation re-visualizes in powerful ways the reality of text as palimpsest of writing and reading that is palimpsest is a text that's written on top of another text right like a blackboard that's been written on in their race written on etc that is reading creates what I call ser-texts strange interfaces or frontiers where new roles and meanings are fabricated thus confusing dichotomies like creator versus reader creator versus interpreter host versus parasite you know because this is a very common way in which we think about texts in at least in literature or in schools is that the students are kind of the parasites on these texts which are the hosts and you've all seen alien or how many of you are watching fortitude the new series don't you watch like yeah who the hell where am I anyways so just to give you a really what is going on with this just to give you a brief example here's Edna St. Vincent Millay a very famous American poet and she has this sonnet called love is not all and it's basically an anti-romantic sonnet which is quite radical given that you know most sonnets we think of as being very romantic and especially given that it's written by a woman especially that it's written by a woman you know in the first half of the century so I asked my students to visually annotate the poem that is a throw up images that somehow respond to the poem capture their interpretation etc. I just want to draw your attention for the moment to the one on the right all you need is love false the four basic human necessities are air, water, food and shelter and this is a fellow from that show The Office isn't he what's his name I can't remember his name Dwight, this is Dwight from The Office right so all I want to point you to here is look at the really interesting intersection of three different historical and textual moments here right one is The Office with Dwight which is a television sitcom from what the 90s the 2000s the 2000s thank you the other one is The Beatles right isn't that The Beatles all you need is love doodaloo right and that's what from the 1960s or 70s I don't know okay and then we have the same the Malay poem which is from the 1920s 30s right and so this is a kind of annotation that works as an inter assembles an intertext and what so what you're like whatever you know it's cute these memes are all over the place whatever but I think one thing this can help in the classroom for instance is to show how the idea of love is a cultural trope you know love doesn't mean anything except how a culture defines and writes the stories around love whether that's Dwight whether that's The Beatles in the 1960s whether it's Malay on the 1920s or 30s so that intertextual the surtext that's created here is really a surtext that is concerned with love as a kind of cultural narrative or cultural trope and the historical meanings thereof I'll just show you one more which I happen to like a lot I just want to hear Sierra Moreno's visual annotation of the love is not all in which he really picks up on the anti romance and presents this woman image really casts the poem kind of as a you know dangerous or the woman taking up this kind of dangerous position in relation to love and what not alright let me move on to my third one online annotations written text to a whole new ecology of signification visual and sonic creativity thus invoking new meanings and pleasures and subverting the notion of text especially literary texts are monumental or transcendent the experience of meaning is always profane and historical not canonical and timeless so again this is a William Carlos Williams poem to LC which is a really wonderful beautiful poem but the point is again I asked the students to do a kind of visual as well as kind of intertextual annotating of the poem etc and you can see that what they do is they in order to make sense of the poem and to explain the poem to themselves and to their comrades they draw on all these different context right visual context literary cinematic etc and so many of them are not it's not Shakespeare they're not referring back to Shakespeare right they're bringing this poem into a context which is quite either you know typically quite contemporary and also popular so these annotations also represent a sacred the transcendent the literary comes into contact with and bathes in the profane the common and the multitudinous every day kind of culture and that's really important for the students in terms of finding a way to inhabit these poems the realm of literature and culture etc okay and finally online annotation and acts interpretation as a social not as a gift of authority text and writing or means to conviviality and community outposts of a cooperative commonwealth rather than exercises and subordination and authority and this is Ginsberg's howl which of course was first performed here in San Francisco many many four to five decades six six oh my god six decades now and I'm just putting this up here to give you a sense of how the students when the students work with texts or you know materials through annotation the one thing that happens on the border between student and text is that they realize that texts are not something to be you know to kneel before the texts are not something that are powerful and nor are the people who are in charge of those texts like teachers powerful priestly etc but in fact that you know they can play as much of a role in understanding and meeting those texts as a teacher or delegated authority etc okay and finally I'll just close with this I have a whole thing in here about unlearning that students need to unlearn as much as they need to learn but I'd like to close with this quote by Kenneth Burke who is a somewhat famous more famous than sleep dealer American American cultural critic who never lived in a haunted house but anyways this is a quote that Kenneth Burke used as a metaphor for learning which I happen to like a lot quote imagine you that you enter a parlor you come late when you arrive others have long preceded you and they are engaged in a heated discussion a discussion too heated for them to pause and tell you exactly what it is about in fact the discussion had already begun long before any of them got there so that no one present is qualified to retrace for you all the steps that had gone before you you listen for a while until you decide that you have caught the tenor of the argument then you put in your ore someone answers you answer him another comes to your defense another aligns himself against you to either the embarrassment or gratification of your opponent depending on the quality of your allies assistance however the discussion is interminable the hour grows late you must depart and you do depart with the discussion still vigorously in progress that is I think one thing that the web or online annotation has helped me to do is to make my classroom more like a really kick-ass cocktail party which is what Burke is describing thank you thanks again Larry alright hi everybody I'm sorry for my preposessant Kathleen Turner voice today I have a bit of a cool butt I will do my best here so yeah as he said I started doing annotations in the classroom about nine or ten years ago eight nine years ago and my students you know we would gather around discuss this chapter of the book and the reaction was much like you know have you seen alien and you guys all blink blink blink no and so I was really frustrated and I thought you know I have this student base that is not necessarily terribly motivated to read or engage with the text naturally but they are terribly motivated to get a grade from me and so I thought okay I can at least get them to get a grade for their annotations as motivation and then that would fuel our discussions and as some of the other panelists have stated already I now use those annotations to personalize the discussions that we have I can easily grab them whether they are digital or flip through their books and find something to actually address something it drives the lesson it makes it student centered so that they are truly leading what we are talking about and it also sort of pushes them to think you know about the surface level when you are talking about high school English even I teach AP English composition and even those students they have to be really pushed to get into the deep analysis of it rather than just the surface level stuff so it sort of pushed them to do that the other thing that it does I feel for the public school classroom the big buzz word is that it differentiates itself and so students at that level are always encouraged to be differentiating for our students whatever level they are coming at us with and that can be terribly challenging when you are pulled in many directions with 35 students every hour but reading and annotating a text can I think be it naturally differentiates itself students who absolutely do not understand something can start at the very basic level of summarization and students that are actually almost in this college level sphere are working more at deep rhetorical analysis of diction and tone and what not so I started very very old school in the public school classroom even here in the Silicon Valley and all around the Bay Area you will be shocked when you enter a public school classroom and find the lack of technology I work in a pretty affluent public school district and have for quite some time and even they do not have one to one devices so look at this pile of annotations at the end that they pulled out if they will not digitally annotate for me or if they won't purchase a book for God forbid they end up in having all these post-its at the end which I cry about when they pull out but okay so one thing that I found when I moved to digital annotation was that it became sort of this living document that everyone's talking about and that it increased their engagement and that they found like they were a part of it and a huge part for me also is that I don't like teaching things that are not relevant today which I think is rare but high school students often need to be led to see the connections nothing kills me more than when they say this is so irrelevant I don't understand how to kill a mockingbird has any relevance today and yeah so I started a while ago with notability I don't know if anybody remembers that and I would have them annotating everything on there and then I started getting really desperate and asking them to snapchat things take pictures of it put your comments on it using Padlet in the classroom so it would sort of start to incur like devolve with our discussion I've used a couple of other platforms Slate has done a couple of really cool things they've already annotated text you know of course Genius has proof rock on there and lots of other things so I'll sometimes ask them to sort of engage with things that have already been annotated and then the previous panelists talked about how they're taking doing visual annotations which I really love as well but another big thing that's really important to me is bringing in the current news, something that's current so my students are reading The Immortal Life of Henrietta Lacks and they're connecting this with sort of the ethics of science that is happening today and these are things that they've annotated with images and with links to other articles and then this was with this is using Hypothesis to look at a satire from Twain right before getting into Barry Finn so it does help I think with the students with engagement I think some things that when you're thinking about developing things for the K-12 classroom you really have to think about how fast can it be implemented how is there sort of an analog version that you can like quickly go to if the digital is not going to work I think you know is it cheap is it free I often times have my students the only thing that we have in class to work with would be their phones so if it has an iOS component they're more likely to be able to engage with it on a regular basis and I think as long as it's something that is really getting them to engage with the text the richness of the text and it's not stripping that away it's not providing them for kind of like an easy out way to think about it go deeper we in the secondary classroom I think are very interested in that so that is all for me so my name is Brian Luca from the Co-Founder of Perusal this is a collaboration my partners they also teach they are in the physics and political science department at Harvard Eric Mazur, Gary King, Kelly Miller we all sort of come to this from a similar problem which is how do we get students to actually do the reading so I want to talk about the problem a little bit talk about what we've developed is to solve it and then talk about some research we've done into engagement and effectiveness of the platform in helping students engage with the reading and also improve their learning so really the problem that we find is that we assign reading to our students very carefully selected we're hoping they're going to do it before they come to class so we can spend class time on the really tough stuff and students just don't do it and sometimes they don't do it because they don't think it's important they don't think it's a necessary part of the course they think they can just come to class and get by without it sometimes they intend to do it and they start working on it they get stuck somewhere in the room and they're trying to understand this complex material and sometimes there are some students out there who just don't do anything there's no great associate with that which one also found so what we've done is to try to build a platform to motivate students to read in a couple different ways one we want to leverage the social to get students better motivated students enjoy interacting with each other just like any human beings do but in particular students enjoy interacting online socially so we've tried to build those sorts of social features into this platform that's built on annotation we also want to motivate students extrinsically with grades but with the realization that it's very difficult to do that at any kind of scale so what we've tried to do is to build a platform that is accessible to students doesn't require a lot of training for students to understand how to get started and is also zero overhead for the instructor so most of the instructors that we work with through our platform are not necessarily interested in annotation per se it's more of a means to get students to come to class prepared and so those folks don't necessarily have time or the inclination to want to participate in discussions and go through things online or do grading of student work so we need to develop something that was completely automated and very with a very low barrier to entry and so what we've tried to do is to build something based on annotation because we know how well that can work but strip it down as much as possible so it's very simple very simple for students to get going with this is what it looks like from the student perspective so from the student perspective you're seeing conversations these are annotations and there are annotation threads just like in hypothesis or any sort of annotation platform but it's completely stripped down just to the text there's no tagging or grouping or anything like that just it's a conversation to the students it feels like chat and we try to make it as much as possible feel like a social network so students can see if you look at the upper right they can see the avatars of who other folks who are logged on at the same time certain things other people say will show up as little talk bubbles coming out of that person anything that their peers do while they're reading will show up in real time so when students are online the night before trying to prepare for a class the next day they'll see stuff appear in real time all they do is just select something let them else go a conversation will start to say to do anything else in particular just type something in hit enter it adds to the conversation and we've also tried to add some things to make it feel like a social network to students so if you look at the thread on the top you're seeing that orange question mark so that flagged as a question and if the student clicks that question mark it indicates that they have the same question as the person that asks that question so in some sense students are upvoting questions if there's a particular response they like they can click that green check mark on the right which indicates that that particular comment was really helpful and helped their understanding so to help their peers try to zero in on comments that were particularly useful and so what we're trying to do is to put students in a mindset of this is your space this is a place for you to help each other better understand the material so what we say to students is look as you're reading if you see something you're confused about highlight and ask a question if you see something that a friend has said and you think you can help answer their question or elaborate on it then respond to it help them out so we're trying to build a community where students are really helping each other all this works similarly to how a social network would work so if I ask a question and someone responds to it I'll get a notification of that I can get an email notification I can reply to that email within my email client that reply gets posted right back to the right thread within Perusal I can mention a friend in my post that friend will get notified we're trying to bring students as much as possible back into the text as many times as possible so they're not just reading and doing some sort of activity once they're coming back and engaging over and over again with this stuff both by themselves and also together with their peers so this is what it looks like I can do fun little things like put emoji in which students like to do I can put some math into my comment hit enter it gets posted we try to figure out if they're asking a question or not and tag it as a question if we can it's an annotation platform we try to streamline as much as possible and add social features to make it feel like a social network this is also self moderating as I said at the beginning we don't want to send the instructor a bill for their time by adopting this platform so students moderate the platform themselves a student can flag a particular comment if it's inappropriate or if it's plagiarized from somewhere that comment gets hidden from the other students so instructors don't need to be constantly moderating or monitoring the platform what we then do is we aggregate all of the comments and questions together to give the instructor a sense of what's happening with their students what are the things their students are asking about what are their students confused about most instructors, particularly instructors who use this in a large class don't have time to sit down and read all the comments and ask questions that students have made it's just too much data so we have this what we call a confusion report which is automatically generated and it clusters the different questions together that students are asking and gives the instructor a sense of what should I be focusing on in class so what you're looking at here is a sample confusion report you can see the questions have been clustered into various topics that are related to questions and within each topic we give the instructor more questions that exemplify the kinds of things students are asking about in that topic and those little numbers a little hard to see but to the right of the question those tell the instructor how many other students said this question was one that I really want to know the answer to and so what we encourage instructors to do which can be really powerful is to take the questions from the confusion report pull them into their own their slides when they're going to teach in class the next day and actually frame class last time around these questions so come into class say to students hey Emily or whoever or maybe even not even name the person had a great question in Peruzo last night you know she asked XYZ now let's talk about it let me I'm going to give you a mini lecture on it or maybe I'll do an active learning activity but it creates a really nice positive feedback loop for students because they start to see that the questions they ask will actually influence what happens in class and motivate them to read more deeply as well and from the instructor's perspective they don't need to monitor things they can just pull up this report as they're preparing for class to keep students accountable and to get students to do this the social is motivating and that creates an intrinsic motivation but we also as I said before we have an extrinsic motivator as well which are the grades that students get for doing this work is not just to account for the volume of what they're doing but also account for the quality so the quality is fully automated scoring so we've developed a machine learning algorithm that automatically assesses the quality of what students have written for their questions and their comments and we've actually found is that we can give scores that automatically that agree with the humans scores as often as two humans would agree with each other so the scoring actually matches quite well with the scores that a human would give so we grade each comment or question automatically based on the quality we also grade based on the quantity so typically instructors will say I need you to make at least three comments or five comments or seven comments on this assignment and they can be questions they can be answers they can be elaborations we can set a deadline we can give credit for late work and then we also look at the distribution of the comments across the chapter because sometimes what will happen is students will put in a lot of work on the first few pages of the chapter and then kind of drop off and just skim the rest so through the grading we encourage students to distribute what they're doing throughout the whole chapter so they're really engaging through all the material so let me just close by sharing some results that we found in our research about how well this actually works so we did a study where we had a physics course at Harvard the students used perusal for two consecutive semesters in the fall of 2015 in the spring of 2016 so what you're seeing here is a graph of how many assignments the students missed one of the things we're trying to do is at a base level actually get the students to do the reading so here we're just noting that almost all the students did almost all the assignments typically students are going to occasionally miss one or two reading assignments for various reasons personal reasons but the vast majority of students are doing the vast majority of the assignments and just as a check because sometimes Harvard students aren't necessarily representative of typical students we also had a course at University of Central Florida that used perusal in the spring of 2016 and we have their data here is that third bar and it's a similar pattern so we are really getting students to do the work whether they're at Harvard or Central Florida or wherever both the intrinsic and extrinsic combined are really motivating students to actually engage and do the reading and they're not just opening it up we can also track through perusal how much students are actually reading through the entire chapter and they really are so what you can see here is the vast majority of students are doing either all of the reading or really the vast majority of it and very few students are just reading parts of the chapter so that gives us a lot of confidence and then let me just close with an assessment of the impact on learning outcomes because that's really our real goal we're trying to get students to do the reading of itself but the main point of it is to help students better improve their understanding of the course material and we find that it really does what you're seeing here is comparisons between the course where we used perusal and a previous version of the course that also used an annotation tool but not this new software that we developed so the light bar on the left are showing the version of the course where we used a different annotation tool the dark bars are the version of the course where we used perusal and in the fall and the spring we had five exams in each course and there's a pretty consistent pattern of seeing improved exam scores between the first iteration and the second iteration of this course which gives us confidence that we're seeing effectiveness not just in the fact that students are actually doing the reading but this is carrying all the way through to their exam performance indicating it really does help their understanding of the course material so let me close there I'd love to talk with you more about this or about any kind of annotation here's my contact info so feel free to stay in touch I think we've got a few minutes maybe for questions if there are questions for the panel so it looks like Nate's got a microphone I just want to come on up welcome to ask away looks like we've got a few people hey guys thanks for the great presentation so I've looked at doing some pilots and stuff with schools but I bumped into at least below the college level there's some legal requirement not to put student work out in the public can you guys just tell all of us about that so we can know what the real story is when we try and do what you guys are doing actually I had a very similar question our general council has actually asked me to figure out how to handle the copyright not the student record the FERPA part for American and I know you're probably teaching in Canada so but for the American teachers you know we know what our exceptions are for in classroom use but the minute it leaves the classroom and it goes up on the web or on any platform so I don't know if your question the previous question was FERPA so copyright in other words is it a student record and why are we sharing student records with other students but my question was about the copyright part because we're not as worried about the FERPA and we actually the general council actually asked the library to create an agreement because the IP policy for Caltech would say the students own their copyright so the exceptions in American law would say as long as it's in classroom use we don't need to do anything beyond we would need them to either license their work under or wave their rights or do a some kind of a permission so I don't know if that relates to the prior question where are you copyright in FERPA okay copyright in FERPA at least if you're in the US yeah I mean I'll just say really quickly about your question at the secondary level I would imagine that it would have to be some kind of a student waiver the same way that if you take a photo of them they have to give like their parents have to create a waiver but I can't give you like a specific ed code answer to that it's way above my pay grade yes maybe I can comment on the second question about content so we've intentionally tried to design our platform as a closed system in the sense that any comments that students make are only shared with other students in their class they're not publicly accessible we what we do to get content to the system is we work with publishers to deliver their e-books through our platform so students can buy the book through us and then access the book within our system and use all of the social commenting tools and instructors can also upload their own materials but all of the materials that are uploaded by the instructor are only viewable to others in that course so the instructor is not uploading things to a public space so that's how we've addressed those sorts of issues in Canada we have other the regulations are a little different but we have for example all these rules around not hosting any student content or any student information on US servers everything has to be hosted within Canada to keep so that you guys can't your government can't come in and just request any information about the student and so but my approach to all of this has been to if I was to take this all into a private space it would undermine all of my motivations for doing this work which is about having the students be engaging out with the public and so I don't have and I somehow maybe slightly purposefully haven't inquired too much about what are the actual obligations and what I tend to do is just I ask the students to do this and they don't they haven't pushed back on putting especially with the annotations some of them push back around publishing their content their essays publicly but that's because they are embarrassed about what they write and so then I try to tell them that if they don't feel good about it they shouldn't be putting it out they shouldn't be submitting it to me in the first place but I don't know the actual answer but I think it's important for us to try to push those boundaries a little bit because if we are having the students not engage with the world then we're undermining the public role of universities and the public mission of universities it's about having the students engage with content that's out in the world that's for me the most fundamental thing and so whatever the regulations that would be there to be a barrier to that I would want to push against well I actually have a question if you don't mind did you want to answer that one too Raimi let me just very briefly say so at the K-12 level with FERPA concerns that's a significant issue that I think is one best navigated in conversation with Jeremy I don't want to put him on the spot but I know that Jeremy does a lot of work as SVI with K-12 districts and there are many ways to respect what is legally obligated as well as district policy and we have in some work that's been done work at either individual schools or district wide where there are partnership models in place to ensure that K-12 students can safely and privately engage in these rich annotation conversations and do so within the bounds of what policy they are legally obligated to follow so it's possible but again it's I think a deeper conversation to go into just wanted to mention that so a lot of you spoke very eloquently of how annotation has enabled your students to engage much more and connect more deeply with texts you've also in several ways talked about how annotation has enabled the students to engage on a human to human level more deeply both with you as teachers and with each other as students and so I know earlier it was yesterday at some point somebody asked about the human to human changes and connections that are made through annotation as opposed to the human machine or human text connections and I'm wondering if if you would either place a significant amount of the value of annotation in what it does at the human level between humans or if you primarily see the value as being between the students and the text itself or maybe it's not either or there are many human to human relations in a classroom obviously and certainly one is between the teacher and the students so the annotation is extremely important to me in terms of getting to know that student in terms of me you know for instance using the annotation in the classroom to let the student know that I know what he or she is thinking and reading about etc or to ask the students to lead the class you know what I mean like if there's a really great annotation on a really difficult poem I'll say hey Maureen you said this so that's a human to human I think in terms of the student to student kind of relation it's a little bit harder to put your finger on I think the use of annotations or at least my experience is that it does create a real sense of collectivity amongst the students on the page so to speak that then you know what I mean really changes the tenor of the classroom as well and I don't have any data or what not but I can say before kind of using online annotation it took a while to get the students you know past the kind of I'm looking at you teacher not at the person to the left or right of me and now with using the online web annotation or what not that community-communal relation really I think gets going a lot faster and becomes deeper I mean I would say also students can be really hesitant to speak up sometimes and to be not exactly the expert but to voice what they have to say in opposition in addition to what you have to say and so it kind of starts a chain reaction of okay we have some knowledge here we're building some knowledge when they see that each other they have something to say about it it's a little bit of like a piling on effect there I mean there's also talking human to human there's also the generational benefits they're going to see connections that we don't all see just from cultural touchstones and things like that that bring some of the connections that he was talking about in terms of the images and all of these cross-cultural connections that they're making to the one text I think that human-to-human connection is the most important thing that we're going after one of the things that we now suggest to instructors is to avoid jumping into the conversations that students are having it can be tempting as an instructor to look at what students have been writing and say wow there's a question I could answer that question really well let me just answer it right now maybe you answered the students question but once I start jumping in now other students may be more reluctant to jump in because they're going to wait for me or for the instructors to respond and so letting students keep the space as their space I think helps foster those sorts of connections I have a question also about the involvement of humans but yeah so Juan Pablo and Brian both mentioned different methods for grading quality and kind of assessing the quality of annotations so yeah I wondered if you could speak a bit more about that specifically like how the human teacher fits in I mean I have an approach to teaching which the students sometimes push against a lot because I try to not lay out any specific parameters about anything and so it's my I don't tell them how long the essays have to be I don't give them a minimum number of annotations that they have to make I give them a set of 10 different kinds of things that they might want to annotate about so you might want to like I tell them you might want to define a term that you had to look up and just put that definition so it's there you might want to link out to some additional resource you might want to ask a question you might want to answer a question so I give them some very broad parameters and I try to not prescribe what it is that they should be doing on that text the students sometimes just want to be told how many is the right number of annotations they need to be, how many words they want to know how many links they should be and how many reference they want to know all these, even at the graduate at the master's student level which for me is incredibly frustrating because I want to give them the creativity and I'm trying to open up a space where they can be creative and where I'm then willing to value all kinds of contributions that I might have not anticipated and so this is where I've been trying to think about what are different ways for me to be feedback and in terms of what so I continue to encourage things that I think are making a valuable contribution at the end of the day that what's a valuable contribution ends up being a subjective thing that I'm sort of passing judgment on as I'm looking through I'm like yes this is contributing to the discourse or look this generated some effect and so it's something I'm still even though after three years I'm still struggling to figure out what's the way of communicating this to the students but in this keeping things vague but still communicating both expectations and then having some sense of fairness around if I'm going to pass judgment because I do need to assign a grade at the end of the day how do I do that in a way that it's not an affront to their to their sensibilities and to their perception of what they were doing I know that I don't think that really answered your question but I hopefully expose a little bit of how I've been trying to think through and approach it it's a real challenge and this is what I'm struggling with as I'm trying to get better at being a teacher is I'm really struggling with how do you leave opportunity for doing things differently and not prescribe how things need to be done while still motivating and encouraging certain kinds of certain kinds of behaviors or certain kind of valuable contribution I don't I haven't learned how to do that yet I probably need to spend a lot more years being a teacher before I feel confident that I can be the person to answer that question I'm going to jump in as well if that's okay so I also teach using hypothesis regularly in all of my courses now and I want to answer your question by talking to the fact that I think the nature of annotation as a practice particularly as a social practice changes over the course of semester let's say and it also changes within the discursive practices even in a single text and this gets in some respects out there you know you're really kind of interesting comments around this border and the definition of what is the context and what is the activity that is on the text so here's a very practical example I teach a doctoral level learning theory seminar and my students as the semester progresses will find different ways of utilizing annotation within even the same text and I see three kind of dominant purposes of annotation particularly as they have more sophisticated learning practices over time so in one respect they're just working on comprehension what do these ideas mean this is a class about learning theory we're reading Vagatsky we're reading Piaget and we read more contemporary ecological perspective this is rather complex work at the PhD level and at some point annotation is used entirely around understanding concepts and relating those then to other theories scholars what have you at a second way of annotating and having conversation these are also future researchers and future scholars and they should be pushing back on other research and sharing their ideas in perhaps a more healthy critical way and so we see annotation as a way of speaking back to the perspectives of authors what's missing what's not here who's not been included and that idea of public critique in conversation with a text and author is an entirely different way of annotating than simply reading for comprehension and then the third way which is also very distinct again comes because again these are future researchers and they're reading literature so that they themselves can write and so they're highlighting and annotating and extracting information for a future literature review or to support a claim in an argument and so as my learning community evolves over the course of a four month semester I need to be responsive to this kind of evolution of annotation practice that really meets very different objectives so if I'm aware of that then I can maybe perhaps look at the quality and grade that but I need to also then be responsive to the kind of annotation needs of my students even within the same text the last the other kind that I see a lot of is the students just trying to entertain each other and talk to each other in conversation and then I tend to see that as a valuable contribution right they're building a community of practice with each other they're not really enhancing their understanding of the text but they're helping to keep each other engaged with the ideas of the text so they're they post memes for each other and they're like making they start sort of just harking back at like they make inside jokes that I have no idea what they're talking about so you know the program I teach is a cohort program they're together a lot for eight months and so by the time they get to the end I you know they're laughing all the time and none of us instructors have any idea of what it is that they're laughing about and that's coming in and the annotations on top of all of those other things I think it's a valuable contribution but even though it's not part of learning what the text is about just to add to that original question I think when it comes to evaluating these annotations for a grade I think this also sometimes goes back to that tension that you get in a humanities course where there's an art there's a level of style that you're being rewarded for and students often want it to be a science and they want it to be you know I did this so you know give me the points and so I am curious how your platform is weighing in on that in terms of giving it some sort of an assessment score sure yeah so what we do is we first of all we look at the grades that come out of annotations as being different than the grades you would give students for an exam or for a project or for a paper in a sense that those sorts of grades you're looking to use them to differentiate between your stronger students and your weaker students so you can you know come up with an end of semester grade this may contribute to the end of semester grade as well but in a small way and the main purpose of it from my perspective is really to motivate students to do what we need them to do to be better prepared for class and for the discussion that you're going to have in class and so to that end we've set up a scoring so that it's it's we've tried we try not to be over precise so each annotation is scored either on a three point scale so low, medium, or high and that we found that those are that's a schematic that we can classify annotations into pretty reliably when compared to how an expert teacher would manually grade the same things Hi, I'm George Corsair from Second High Valley State University I'm a fellow young faculty member and I have a question that's basically the distinction between a technology and psychology we just heard it, you guys started talking about it while I was sitting here in line somebody told me if you want to ruin the enthusiasm keep it up like this like that, okay, wow this is pretty good if you want to ruin people's enthusiasm put the topic in the classroom in other words, students annotate they annotate the heck out of Facebook they annotate the heck out of a lot of places, right it's very similar to annotation anyway it's not annotation itself the question is what is it that they're eager to comment on what are they eager to think about they want to think about the person of the opposite sex sitting next to them in the classroom they don't want to think about, you know, Shakespeare right, that's just reality and in my classroom there are a few, but not very many who ever are going to be researchers they're mostly people who are going to go out into you know factory jobs or if there's any left or something, you know, farming or whatever it is but it is important that they play the role of a citizen it's important that they learn the things that an educated, modern person in a contemporary society needs to learn and I think annotation would be good the question is how do we as educators divide the stuff that we can fix with or help with in terms of the technology of annotation versus the psychology of just human beings that are going to screw off and be mischievous and be normal people at that age, which is cool but may not be appropriate in all cases but you know what I'm saying we have to mentally, how do you separate that? How do you deal with that fact and not conflate the two things and saying, well jeez you're not just the most wonderful student you could be so much more, you know what I'm going to work at my dad's shop when I graduate, right? Well, if I can answer that I think one of the interesting things about annotation is students don't know what they know often times students underestimate what they know you know if you take that meme the student posted up there from the office you know they kind of threw that up there because they had some idea about love and being cynical about love but really there's a lot in that meme and a lot in understanding that meme and understanding why it's significant that they don't really know I think the other thing to think about is this is we think, you know and this is kind of problem with the technology as well in a way and also with our this continued dependence we have on the page I think as a kind of vehicle or medium but in any case we think of annotation as somehow bordering or bounding a page I think sometimes that it's like footnotes or marginalia but what if we thought about annotation in a different way that is every time we annotate a text we are taking the text out of where it belongs and putting it somewhere else whether it's in a conversation or in the case of some of the visual stuff I do where I'm kind of asking the students take the text out of the classroom and put it in TV land or put it in movie land or put it in music land from the places they're supposed to be they can be very impious in a way taking text like Shakespeare or Malay out of the classroom which can be much more interesting, lively active and important for students yeah I mean I think a lot of times it just comes down to making the personal connections for them and that's going to come out of getting to know your students those things come out when they're writing about the personal connections that they're going to annotate and also it's allowing for that so letting them know that their personal connections are important to it and that it doesn't have to be completely high-minded rhetorical analysis so you know it just comes specifically in terms of what you're asking them I think yesterday I asked my juniors you know you're going to set J. Alfred proofrock up on a date what are you going to tell the other person and that's what they annotated and they all told them to get therapy but you know okay and I think we should probably just take one more question and wrap it up so reflecting on my experiences when I would participate in classes I want to ask a question about is annotation itself making the classes richer or is it merely the fact that it compels people to actually do the reading I think it's both I don't think it's one or the other I think that the fact that the students have done the readings it enriches the classroom right this is like even if they weren't annotating if they could just commit to actually if they were reading it through to the end and not annotating the class discussion would be better than when only half and one of the most frustrating things is when you have three four people that have done the readings really well and then you have another 15 people that haven't it you end up spending the class sort of explaining what the readings were for those 15 people and then those people that have already know what they were are just they're doing they're doing nothing or providing that explanation but I think that the annotations themselves like I said I think for me it's been really important for me to be able to pull out what the discussions were and then you can take the conversation so much further then because already all of what would have happened in the classroom would happen already in the annotations now you have an opportunity to have a completely new kind of direction to take the discussion in and I think that that's the kind of enrichment of the experience that I think is the additional value of actually annotating beyond just compelling the reading in the first place as you can see from my hairstyle I've been around for a while and I've been around long before online annotations and so in other words what I'm trying to say is there are eight million ways to make students read if you want to make them read and those have been around since long before online annotation and so really for me the issue is not that because I can solve that problem if it's a problem in the bright ways it's in a way what does it mean when the students read there's reading and then there's reading there's reading like yeah yeah Hamlet some dude who has some father issue or whatever then there's to dream or not to dream so there's different kinds of reading and the goal is not just I don't if the student doesn't read or if the student just reads it like they read a Chinese takeout menu it's the same to me what I want is the engaged authentic real you know what I mean and I think the online annotation etc is a really valuable tool for that yeah I would also add that it kind of makes the student go back to the reading the way that you would as the teacher in the sense that we're pulled in a million directions every day and no matter how many times we've taught that text we need our annotations to throw us back into it sometimes and to get everything going and to make it meaningful same thing goes for the students it's I find that it's a much more meaningful conversation because they're not losing all that critical thinking that they were doing you know the first time around when we come back together as a class really briefly I think also what we found is that a lot of students don't know how to read they know how to read but they don't know how to read right and so to me one of the things we can do with annotations is model active reading and show them this is how you can really get a lot more out of reading than just passing your eyes over the page understanding all of the words can we have another round of applause for our panelists please and thank you for everyone's