 Welcome to this Hypothesis webinar on research on annotation in English and composition. I have with me Alan Reed from Coastal Carolina University, Michelle Sprouse from the University of Michigan, Noel Braithwaite. Did I get that right, Noel? Brothwaite from SUNY Farmingdale and Julie Severe from Southwestern University who's in the same general area as me. I'm Jeremy Dean in Austin Texas, the director of education at Hypothesis. Those are our presenters. I'm going to offer a brief introduction and then we'll let them take it away and share their their research on annotation in English and composition. But first I'll intro about hypothesis. Actually I want to first start off with, if I can advance my slides, a quote that was shared with me actually by a collaborator of Noel's. Mary Tracester is at USC, who's also doing research on annotation. She wasn't able to join us today but she introduced me to this quote from Robert Scholes that I kind of want to just open with as food for thought as we discuss annotation in English and composition today. So Scholes writes, we normally acknowledge however grudgingly that writing must be taught and continue to be taught from high school to college and perhaps beyond. I know that personally I got a PhD English and taught comp for many years at the University of Texas. We accept it I believe because we can see writing and we know that much of writing we see is not good enough but we do not see reading. We see some writing about reading to be sure but we do not see reading. So that's just something I want to file in the back of your heads as we have our conversation today. So this is brought to you by the Hypothesis project. We are a non-profit, an open source software company developing collaborative annotation software for the web. Of course annotation has been around for centuries probably since at least the invention of the book. It's probably not new to anybody who decided to join a webinar on research and annotation in English and composition. I taught high school and college for many years and really spent the first day of class every year trying to inspire my students to write in their books as a critical practice for success in my class and for literacy and engagement more broadly. But again like Scholl says I never got to see that that work of annotation. It was in their own private paper books that they were taking home every day. Maybe once in a while I'd stroll the classroom in a seminar and peer and see that somebody was indeed writing in the book. I actually had colleagues that would show you know for students that this was in high school open their books and show that they had written in the margins but even that even though it was evidence that it had happened you know how it was being practiced how it was being done was not really delved into. So what hypothesis is trying to do broadly is bring this ancient technology of annotation that you know scholars and students throughout the ages have practiced and bring it online and make it part of the fabric of the internet not just in education contexts you know for classroom use and not just for scholarly use but really as a basic practice of everyday citizenship on the web. That is if there's a layer of annotation on top of the web the sort of critical practice of reading closely and thinking critically that we teach in the humanities and English and composition especially becomes part of how we might interact with with the world with information of the web and that has profound implications. It really raises I think English teachers and writing teachers to the most important role in the curriculum or one of the most important roles in the sense that it'll matter how we're looking at information and discussing information online moving forward. But hypothesis does have great traction in education. I'm the director of education as I said and most of our users are students in classrooms both in the K through 16 and high school and college levels predominantly and there's sort of three takeaways that I've gotten from my experience working at hypothesis and before genius and the role of director of education. From feedback I've gotten from from teachers from my own experiences using annotation in the classroom and one of them is that hypothesis is a really collaborative annotation social reading makes reading visible so it does sort of subvert what Scholes is saying earlier where we don't see reading now we can actually see the reading right we can see highlights we can see comments we can see the discussions of our students in a way that we really couldn't before. In some ways that could be taken as a new form of surveillance of students right there was an article in the Chronicle of Higher Ed a few weeks ago about what is called reading compliance right we can know that students have read but I think more importantly we can know how students have read and there was a tweet just yesterday from somebody who was interested in incorporating hypothesis and it's the second half that I just want to draw your attention to. I'm realizing I'm grading reading comprehension without really priming it in the class so of course we all want the students to read in our classes right and we expect them to read well but I don't know that we how often we drill down into what that actually looks like and I really appreciated the way that Matt DeCarlo, PrEP Matt DeCarlo on Twitter described that as I can't really grade something that I'm not teaching the practice of so but reading made visible through collaborative annotation now we can really engage in that practice of teaching students how to read carefully. Hypothesis makes reading active and this is a you know close reading practices going back for centuries asking students to engage more deeply in what they're reading grabbing small bits of text and explicating it making sure they comprehend but also going deeper with their with their thinking and analysis and finally I think and more innovatively and I know that Alan and Julia both I believe written about this the idea of collaborative annotation or online web digital annotation makes reading social there are always arguments that reading has always been social people have shared their margin area read aloud to each other for centuries but I think and a piece that Julie has coming out I remember her talking about you know there's something really different here and sort of at least the speed at which we're able to read socially and so that's that's something and this is actually a quote from a student it's one of my favorite quotes from from from a student she published a blog after using hypothesis in the class and she says hypothesis is my literary Facebook when I'm reading I sometimes wonder does anyone actually understand this am I crazy and definitely one of the biggest pieces of feedback positive piece of feedback we get from students is that they're learning from each other through social annotation so these three things you know hypothesis makes reading visible active and social is something that I've noticed as a you know in my role as a director of education but what's really exciting about this webinar which is going to be the first of a series of webinars on research about annotation in the classroom this one focused on English and composition predominantly I'm used to have researchers I hope substantiate some of the off you know anecdotal claims that I've made in my experience or that I've gotten from from users of hypothesis with with actual research study with you know analysis behind it and data behind it so that's what I'm looking forward to here and I will stop talking and turn it over to my presenters I don't think I gave any warning to this about what order we were going to go in so I hope everybody's just prepared but we're going to start now introduce each of the panelists right before they go we're going to start with Alan Reed who is an assistant professor at Coastal Carolina University and an evaluation analyst at Johns Hopkins University he's been using hypothesis for several years in a variety of contexts we've been in touch since early days and I think I've always had a nice little side conversation about his thoughts about annotation in the in his courses and he's written about it most recently he published an edited collection titled margin alien modern context which examines the benefits of individual and shared annotation practices so without further ado I'm going to hand it over to Alan. Thank you Jeremy I appreciate that and thank you for having me again to this webinar it's been yeah quite a few years several years now that I've been using hypothesis both professionally and personally and as Jeremy mentioned I've written several pieces on it most recently last year an edited collection on marginalia in which many of the chapters talked specifically about hypothesis I think it's really the the frontrunner for open annotation and web annotation in general and so let me know if you'd like to read more about that I can give you some more details I can share some things with you after the webinar just feel free to contact me I'm going to try and share my browser here with you so I wanted to talk about hypothesis in three specific ways that I use it firstly professionally so in my role as an assistant professor I you know we sit on a lot of committees one of the committees I'm on is on the online teaching committee and we evaluate fellow professors courses specifically online courses and so how we do this one of the ways that we do this is we look at their course shells whether it's in Moodle or Canvas and we give them feedback on this in the way that we do this is via hypothesis so we actually use the hypothesis tool to annotate their courses their sites to give them specific feedback on how they can improve those sites for themselves and for students so that's the first way that we use hypothesis currently the second way that we use hypothesis really is in a very unique sort of way and that's through our digital badge program at coastal Carolina University so to give you a quick back story on that all of the first-year writing students at the university take those first two course sequence writing courses you know 101 and 102 in the fall and spring semesters and so what we've done since 2014 we designed a digital badge program where students are actually going through individual badges that teach specific competencies with regards to the course learning outcomes and then at the end of that if they're demonstrating proficiency in that skill such as quoting or summarizing or what have you then they earn a digital badge and it goes into a backpack and I'm sure most of you are familiar with how digital badges work but the interesting thing about this is that since its inception since since I developed the program we've baked in hypothesis since the very beginning and asked students to while they read these different things that I'm going to show you on the site to annotate them and to use it to use hypothesis to do so and so essentially how the badges work they go into the program they navigate to the badge that their professor has instructed them to work on for that particular week let's say it's quoting and it's they're being presented with basically a scrollable page of text that that we've written in-house that's trying to give them a bunch of different guidance on in this case quotation so what is it why do we do it here's some examples all sorts of things but the interesting part is that we've used the hypothesis tool the widget because this is a WordPress platform that we built this on we were able to use the hypothesis widget in plug-in and ask students as they read these different things as they read each of the texts for each of the badges to annotate these things and some professors use it as as Jeremy mentioned a compliance check you know basically to see are the students reading it or are they just skipping to the assignment which assuredly I'm sure many of these students are just skipping to the assignments so some professors are using it in that way but others are really using it as a proactive way of saying you know this is important text don't just skip through this and and and we know that we read differently in digital and in print formats and one of the ways that we can sort of anchor these readers into the text on on screen is through asking them to annotate with hypothesis so that's one of the things that we'd like to do there in the third and final way that I use it personally in my courses is I actually use it as a way to guide students while they're reading so if it's an online article for example this one from Nicholas Carr I'm actually prompting them throughout the reading with my own annotations in the text that I've done beforehand I'm prompting them to do different things so for example I might get to a part in the story where I'm asking them to use a cognitive strategy like I am here and saying okay summarize what you've read so far or in this instance maybe a metacognitive strategy where I'm asking the students what do you understand so far what have you not understood so far what kind of questions do you have so I'm actually using it as a teaching tool in that sense that this is an online text that I want students to read but I'm guiding them along the way so I think hypothesis has so many uses for so many different scenarios but those are the three very different ways that I use it personally and yeah I think I think I'll sort of end it there and yeah I'm happy to talk to anybody after the webinar if you want to contact me regarding any of this stuff but let's let's go ahead and move on to the other presenters if that's okay with you Jeremy sure thing Alan quick question is marginalia in modern contexts out yet yes here you go here's a here's a URL in the chat box to it you can check it out but those texts are always so pricey if you are interested in reading about that shoot me an email we can I can get you set up so great and again I gave our panelists no warning what order this was going to go in and I'm looking at my piece of paper in which I ordered them and I can't distinguish there being any rationale for it but because it doesn't conform to either first name alphabetical or last name but in any case everybody's ready so we're going to move on now to Michelle Sprouse who's at the University of Michigan and is a first-year writing instructor and also a doctoral candidate in the joint program in English and Education at the University of Michigan and Arbor for current research focuses on social orientation as a tool for improving reading in post-secondary context so without further ado I'll turn over to you yeah thanks Jeremy for the introduction I'm really happy to be here today to talk with all of you about social annotation in English and composition before I came to the University of Michigan for my doctoral studies I was a middle school literature teacher and that's where I started socially annotating with my students and I've since socially annotated with my first-year composition students and English teacher education students and as a student in my own graduate coursework so I have a lot of experience I think in different contexts with social annotation I'm also really excited to be part of what I see as a renewed interest in reading in composition studies and just the last few years we've had collections like Sullivan, Timbergen, Blau's Deep Reading and Salvatore and Donahue's reading focus issue of pedagogy come out and I see this conversation really extending to across other disciplines like the the Chronicle essay that Jeremy referenced earlier today and I'm glad that scholars and instructors are really starting to pay attention to reading pedagogy and I think that social annotation can be a really exciting part in helping our students to read better in our disciplines. Today I hope to share a little bit about two of my previous studies based on social annotation and how those findings and methodologies are helping to shape my dissertation research. So in the winter of 2017 I was teaching first-year composition for the second time and I was really interested in how my students were engaging with their readings and we started using hypothesis as our social annotation tool. I had done some reading looking back at the kind of history of three C's that's college composition and communication journal and I had identified four broad reading purposes. The first in composition is reading for ideas or what a lot of people talk about as reading comprehension and it's the ideas that students can extract from texts. Rhetorical reading which is a purpose that's really popular in composition studies now and it refers to reading to identify and evaluate an author's rhetorical choices. Critical reading and here I don't just mean critical thinking skills like synthesizing or evaluating ideas but really reading to develop a critical consciousness of power relationships. So those of you who were at four C's this spring we can think about in ways chairs address how do we language so people stop killing each other or what do we do about white language supremacy. And the fourth broad reading purpose that I found in the literature was reading for affect or thinking about if we refer back to Louise Rosenblatt's work how students might attend to their emotional and effective responses to what they're reading. So in my first year composition class with these kinds of purposes for reading in my mind I collected their annotations over the course of a semester for all of our assigned readings and I had at the end I think about 1200 annotations from students who chose to participate in my study and I went through after the course and I coded those annotations for those ways of reading lenses the reading for ideas rhetorical reading critical reading and affective reading. I was looking for the patterns and to try to call attention to what are the strategies that my students are using as they read and annotate in the class and I found that they aligned generally with the the emphasis that I saw in the literature lots of reading for ideas annotations lots of rhetorical reading annotations and as an instructor I was most interested in those annotations that layered more than one way of reading to kind of get at something a little bit more complex in the ways that they were engaging with the text so if they looked at how an author was making rhetorical choices to support particular ideas those annotations were some of the strongest annotations but as I was working through that study and focusing mostly on just their annotations I had a lot of questions about what my students were doing with the text and with hypothesis while they were reading if you've ever annotated in hypothesis you might know that as you highlight text and someone else comes along and highlights that same text the opacity of the highlight increases and I could see places where so many of my students had highlighted the same sentence that you could no longer read the original text and yet what was happening in the margins was sometimes not actually a conversation they might be repeating the same idea I see Julie nodding her head there they might be repeating the same idea but not really engaging with each other and so I had a lot of questions about what are my students doing when they when they log into hypothesis and they're completing the reading they're complying on the surface with what I've asked them to do but I'm not sure what it is they're actually doing so the following year I designed another study of social annotation this one actually I did with some graduate students and a literacy course all doctoral students all who had previous experience teaching either middle or high school English courses or college level writing classes and I watched three graduate students read and annotate with hypothesis we did a retrospective think-aloud protocol so I had cameras set up and screencast software to capture what was happening on the screen and for two of my participants what was happening on paper texts around the screens as well and then I created the videos and I met with them again and had them think aloud for me and talk to me about what it is they were doing while they were annotating those texts and what what sorts of choices they were making and why they were making those choices and that was a really exciting study for me to listen to other readers explain why they're annotating in certain ways and what they're choosing to write down and what they're leaving off of the paper or leaving off of the margins things that I wouldn't otherwise have any access to and there's some interesting findings from that study the first was even though all those participants were doctoral students who we would consider experts in literacy practices and reading and writing they had anxiety about the reading strategies that they were using and to what extent those strategies would be effective for seminar participation or for future writing projects for two of the students there was a lot of reading and annotation that happened away from the screen so they would prefer to print the text and annotate by hand and spend a lot of time engaging with the text before they even got to the screen and hypothesis and started sharing annotations with others and all express positive responses to that opportunity to watch themselves read and to reflect on their on their practices in the retrospective think aloud protocol and so what I'm doing now in my dissertation research is I'm trying to pull from what I've learned about how people interact with social annotation tools and the kind of methodologies I can use to get at what's happening as students are reading and I'm I've started a design based research project in my own first year composition classroom again. I'm planning three major iterations I've just completed the first this term and I have two more in the years that are that are coming up and what I'm doing is I'm I'm annotating socially again with my students I'm collecting their annotations I'm surveying them I'm collecting information about their writing and looking at patterns and their growth and writing and reflective cover letters and all of my class materials and field notes and what I'm trying to do is paint a bigger picture of what's happening in the classroom system that is shaping their annotation practices so not just looking at the annotations themselves but looking at all the things that are happening in the classroom around it and I've also managed to schedule with some of my students some think aloud protocols again for this study I'm not doing retrospective think alouds so that I only have to schedule one meeting at a time with my really busy undergrad students but again it's so fascinating for me to watch them interact with the text and think about what are the things in the tools that might seem easy or obvious to someone who has some technical experience that might get in the way of my undergrad students who despite their reputation as digital natives are not always as text savvy as we think they are what do they understand about the purposes for which that we're reading in class and how are those understandings shaping what they're doing in the margins of the text so I'm really excited to be working on this research and I don't yet have data from this dissertation study to suggest what kind of growth I'm seeing with my students right now I'm still working through that analysis but it is it is really fascinating for me I think to be hearing my students talk about what is happening as they read those texts and just as with my work with the graduate students there's so much that's happening in their minds that's still not captured in the annotations and I think as a researcher it's really important to me to keep coming back to my participants and thinking about what are the things that they are putting down and what's not being shared in that in that visual way so that's what I wanted to share about my current research that's awesome thanks Michelle what what still remains invisible from the from the literacy process despite the fact that we have these new ways of seeing evidence of reading it's not it's not a totalistic very interesting great so next we're going to hear from Noelle Brathway let me advance my slides here who's at SUNY Farmingdale and Noelle is part of a research well Noelle got her PhD at Iowa State University and her areas of research and teaching expertise are rhetoric and composition professional communication journalism and rhetoric of science and technology and Noelle as I understand it this is a very secretive research study that Mary Tracer is also part of I mentioned Mary before I don't know a lot about it they presented at four C's I wasn't able to attend it but the research group which I believe as others involved and then maybe Noelle can tell us more includes I think participants from seven institutions of higher learning and they're looking at how digital their their study is called digital annotation tools in the college classroom and analysis of the impact of hypothesis on student reading and writing competency I know they've conducted a bunch of research they have some data they have some publication coming out and Noelle can tell us that story okay yes um well actually our group right now is made up of myself Mary Tracer at USC and Chris Carvina at Northern Virginia Community College so it's it's a trio right now that's that's really focused in on on getting our um looking at our data and and putting something together for publication but we're we're not so much secretive as we are very in very I would I would say nascent stages especially after um hearing our our two uh first presenters um we are are very new to the tool we we use the tool last semester in the fall and that's what we're um that's where our data came from um and it was interesting Michelle um to hear you talk about the the different ways of reading that you were focused on because we we signaled in on on the the same areas sort of um looking to see did our students um use the tool for rhetorical analysis which um uh our our partner Christine Carvina specifically um was very um sort of prescriptive in um showing the students and going over with the students how to use the tool in that way um but we were also looking at um sort of effective responses as well as comprehension um and as well as um really the the synthesis and the evaluation that you weren't necessarily looking for Michelle um so those those were the kinds of of things that that we were looking at um from the data to see how our students were using the tool but we very quickly uh discovered when we when we started looking over um our our students responses and sort of um placing placing responses into various categories that we really honestly all three of us had much much um much stronger emphasis on on sort of what we were hoping to achieve in our individual classrooms so because we have different um student bodies and and um this was all for first-year writers um but our our students were uh we had different sort of goals even even among the first-year writing um um course so for me I actually taught a blended class um which so half of my class um traditionally probably would have been in a developmental writing class and the other class were placed automatically traditionally in a four-credit class um so my number one goal was actually um I was really most interested in their in their effective responses and I really wanted to see sort of their just very raw interaction with the text of course we um we had we had plenty of discussion about rhetorical analysis um and and we definitely discussed text in that way in the classroom but in terms of how I was hoping they would use the tool I wanted to see comments like um I can really I this is a confusing here for me or um I I was glad to see that the writer uh brought this point up or um things of that nature because that that has has sort of been a struggle that I've had um in teaching this this kind of blended course in the past and also just a traditional frankly traditional first-year um course so I really wanted to just see them getting in getting their hands dirty um and I thought you know that the tool um was going to make this very visible um and that that I I would be able to you know see them thinking out loud um and like you Michelle I was very very interested in in sort of um what they were thinking about their reading as they were reading um what I found was that I I did get a lot of sort of repetition um of the text and at first I thought that this was um this was not this was not a good thing um and this was was somehow negative but but I realized in thinking of the ecology of the classroom um and sort of thinking about how the students had used the tool um in in terms of of larger class discussions and some of the writing that they did that this wasn't necessarily the worst thing um the highlighting of passages um even if they didn't write lengthy responses I I took this as an indication of there and and this this came out to in some of the the answers uh to the survey question um that they were they were really using the tool as a as a way to um sort of reflect um and even if they didn't have a ready response and it's interesting to hear about your experience with the doctoral students who would do offline thinking and delving and writing before they were ready to share socially I think maybe some of that um for first year students especially the develop those who would traditionally be a developmental writing class um maybe um that that what I was thinking was was somehow not good was actually just a very sort of beginning stage of that engagement and and I'm I'm sorry to rethink how when I use the tool again I'm using the tool currently but um really I my lessons have been learned over the course of the semester going through the data and talk having conversations with my partners so in in the future I I will also want to go back in and and maybe do some read aloud protocols um and do a lot have a lot more discussion with the students about their thinking as they're reading as they're as they're sharing as they're writing the social aspect of it um I didn't see my students really answering one another now my partner Chris Carvina um she actually built that into her assignment they had to respond to one another one another's responses so um that was something that she baked in I didn't do that and they really didn't respond I I didn't really get much of an indication that they were reading each other's responses um during discussion I would actually use some of their responses as springboards to to to make sure that the the the exercise the reading exercise and the ensuing discussions and hopefully the rereading that they would do they would keep in mind their classmates responses and they would would respond to their classmates responses but that wasn't something that I asked them to do and on their own they didn't do it so we we right now are just um at like I said I feel at a very nascent stage um of of of trying to get a handle on um how our students use the tool um how we can um sort of uh structure the the use of the tool within the classroom in ways to really capitalize on all the potential of the tool and um yeah that's that's that's pretty much pretty much where we are great and Noel you guys have something that you're working on for publication where can people expect to find that is that am I right about that that you're collaborating with Mary on something to come out in pedagogy or somewhere pedagogy yes that's that's where it will appear we are drafting our first draft is due in early June um so I'm not sure of the timeline from that point on but that is ultimately where it will be published is pedagogy great stay tuned um thanks Noel actually our next presenter has something forthcoming in pedagogy as well uh Julie Severs is the founding director of the Center for Teaching Learning and Scholarship at Southwestern University where she also teaches in the first year seminar program at the time that she did the research that she's going to be talking about today she was teaching literature and writing courses at St Edwards University in Austin where she was the director of the Center for Teaching Excellence there and as I mentioned her research on annotation pedagogy is forthcoming in the journal pedagogy um critical approaches to teaching literature language composition uh and culture so I'm going to turn it over to Julie hi uh thank you um I have uh I want to make my comments here but I've also created a handout with some of the information I'm going to talk about so let me go ahead I'm going to share the link all right so at the top of this page can you all see my it's a google doc there's a link here and that's the link just to this google doc page um so I'm going to turn this off for a second but I'll turn it back on in a moment um I began working on annotation as a result not so much of um an ongoing research interest at the time I was working on other things but some actual pedagogical challenges I was having in my classroom um the project that I'm going to talk about today actually began in 2016 and um it is coming out in pedagogy but it's not coming out till 2021 which is a ways away and um so I'm glad to be able to talk about it here I'm actually going to talk about using it in a literature course um but I think in many ways the challenges and goals I had in that course were similar to what people are describing as challenges and goals in first year composition courses um at St Edwards University where I was teaching this course it was a required component of the general education sequence so all students had to take this they were not there because they wanted to be and um and none of them were english majors so they were all taking a course that was going to be their one and only literature course and where they had a lot of challenges um both with the kinds of reading that we ask students to do with literary texts and with writing about literary texts developing the kinds of arguments we asked students to make about literary texts um and I had um I had a course that was um had been working for me fairly well but I had a couple specific challenges that I was responding to when I started this project one was that um this required course was a topic space course and faculty offered many variations on this my variation was called the literatures of American religious experience and I put together the course pack and shared it through the learning management system almost all of the course texts were shared digitally through canvas and although I told students that they should print them out what I'd increasingly seen was that students were not they were just coming to class with their laptop or often actually just their phone and my pedagogy I've had the privilege um of teaching in in universities with small class sizes so we I don't do a lot of lecture um I spend a lot of time in class working on the texts and I expect that we would have papers and pencils and we would be marking things up and discussing and analyzing and I just had found it increasingly difficult to do that um so for years I've been thinking about how to uh I had made the decision that I didn't want to be policing my students on the printing issue I obviously that would have been one option and I know a lot of faculty who who go that route they just force students to buy or print texts but I really didn't want to get into that kind of relationship with my students I didn't want to be the the printing police and so I decided to let them use their digital tools and to find ways to help them annotate so um so I've been looking for a while before I found hypothesis and just the ease of use was what I had hoped would really work that you could install the widget it would just install a layer of annotation over a text and there really wouldn't be much technical challenge it's interesting to me Michelle that you've been thinking about where there might still be some technical difficulties um we did have one technical challenge but it wasn't major um so once I discovered hypothesis I thought okay that's great I'm gonna I'm gonna try using this to really um solve this problem so that we can do more annotation so that was the first challenge I was trying to solve I really um I really wanted annotation to be the sort of building block for everything we did in the class the second challenge was that my students were really struggling to write um literary analysis it's not like it's not a kind of writing that comes naturally to frankly anyone but it certainly didn't come naturally to my introductory non-majors who were in a class just because they've been told by the university that they had to take one and um and in particular I wanted them to be able to develop their own arguments about texts drawn from their own reading I didn't want to just assign them topics here five different topics to write your paper about pick one but my students in spite of a lot of scaffolding on my part about uh developing um topic ideas and then finding passages that they would build an interpretation around they really struggle with that and so I sort of decided this was my last stand if I could help use annotation to help students develop organically their own arguments about a text then I could keep having students um uh use their own ideas otherwise I was going to need to start assigning essay topics so that was my second big goal could I develop a process by which students started writing about texts and annotations and then they did some additional reflective writing about texts in low stakes writing assignments and then they used the annotations and the low stakes writing to develop an essay and in particular I wanted to see if that essay then became something that was really richly grounded in evidence from the text so they they were actually going back and pulling those annotations into their final project um I got an assist on the design of um the assignments by actually one of the webinars that Jeremy ran through hypothesis there was a faculty member there who had a research project that really had students build their uh literary essays around annotation and I drew on her ideas so um I'm going to switch over to my screen share and sort of show you what that ended up looking like and let's see I need to get there we go so um here are the questions that I posed about the class that I really wanted to think about as I um as I worked on this project so this was a research project based with the first iteration of this new newly formulated class one was a reading question could this approach slow down their reading and build close reading skills um I had a variety of reading skills I was trying to teach them including reading with attention to detail I wanted them to be looking up information as they read we had more than half of the class had readings that were pre 1900 I'm an early American uh scholar by training and I use a lot of early American sources um could it help them begin asking reflective questions could it help them begin recording some of their reactions and ideas while reading the second big goal I was interested in was this writing question could could annotation social annotation help novice students learn to move from reading the text to developing beginning interpretations and then to formulating a full argument and so I was really thinking about very carefully scaffolding the movement between those processes and then the third question was that's that question of the social dimension of this kind of space um I wanted in my experience particularly teaching uh introductory level courses students don't get that much why we do discussion they're not clear on why they're supposed to be learning from one another they really want to listen to what the teacher has to say and we spend a lot of time talking about the importance of um learning from one another and building on one another's ideas and also when they're writing for them to see scholarship and and research as part of a an ongoing conversation amongst um scholars and students so I wanted them to see themselves learning from one another I wanted to see themselves answering questions for one another I wanted to see themselves sort of building on others ideas and so that was a third question and then I just had some of these other questions that I think many of us have had about tools like hypothesis could asking students to annotate text before class increase their preparation for class um would it make them feel like they were part of a more participatory and collaborative classroom um and a final question was you know my students often came into literature courses thinking that you read a text and you suddenly understand what it means that you can sort of um extract the deep meaning of a text uh quickly and I wanted them to see how much everyone in the class was struggling to work through the meaning of a text some of our texts were quite challenging um so that the effort and difficulty of reading was normalized it was not something that students could feel that was you know they were the only ones struggling with I've been uh working at institutions that have a significant percentages of first generation students and I really just wanted to validate that uncertainty and puzzlement and confusion are common I had uh I had a series of um assignments um weekly text annotations they did annotations for every text they read and if we had a text that was in print I actually had them do annotations in essentially google docs we were using box but um and this was an example of the instructions that I provided they could be reference checks they could be questions about the text they could be their ideas or reflections they could make a comment between a text that they were reading and something else we'd read I did have a an assignment that was designed to help with that social piece each of them was assigned in pairs to a discussion moderator role that really um asked them to synthesize the annotations so after all the students had annotated a text the discussion moderators would go through and write a blog post that was shared with the class that synthesized some of the common themes and ideas that the students had made in the annotations and I asked them to hypothesis lets you um link to us to an annotation not just to the text so they were supposed to cite their colleagues in the class when they mentioned an idea that emerged in the annotations they could they could document that the blog posts where there were seven of them across the course and this is where they did low stakes informal writing but a little bit more extensive than the than the annotations and then project one and two I've hyperlinked those if you want to look at how I structured those they wrote an interpretive essay about a text or a section of text and then the final piece was that they would then post public facing annotations on that digital text that drew on the arguments they'd made in their essay so that they became in a way a scholarly editor of the text for anyone who might encounter it in a public space um so that was the structure obviously I had a lot going on and a lot of different things um I went through IRB and got student consent I developed surveys for the beginning middle and end of course and then I did a close analysis of their annotations their blog posts and their projects um after I finished teaching the class I changed institutions and in that process I lost access to some of my data I lost access to canvas so course text that I posted in canvas I couldn't see the annotations on those anymore so I ended up choosing a couple texts to do close analysis of their annotations for just based on text that I had all the data for so I ended up analyzing their annotations on Benjamin Franklin's Way to Wealth and their annotations on Emerson's Divinity School address um a couple takeaways well I want to show you just a few things here um I asked them before class about their comfort level with some of the kinds of reading and writing we would be doing in class and this is a fairly typical example my students did not feel very confident on a scale of one to ten how comfortable would they be at reading complex texts um I include the second example just because it made me chuckle um this student obviously felt quite confident and gave himself tens in every areas but in reality all of my students were struggling with these kinds of skills um the midcourse feedback showed that students although they didn't always love the fact that they had to annotate every single text we read and I decided to go all in on the annotation so every text we read they were annotating um they acknowledged that it was really valuable and they acknowledged that it helped them stay on top of it they liked seeing their classmates reactions they liked seeing how others view the text compared to me so some of this um some of the questions I had about whether they were getting something out of the the fact that they could see one another's annotations were reinforced um and then and you can read some of this further if you go to the the handout and then at the end of the course I asked similar questions again and I'm just including I had a bunch of questions on the surveys but I'm just showing you two here um students overwhelmingly acknowledged that writing annotations helped their learning about the text um 41% said most the time here 53 some of the time just a small sliver said that they felt that that was really helpful their answers were quite different when I asked them about reading others annotations um and you know I I think that this reflects both um their kind of um mixed understanding of why you would even want to listen to your classmates when they're other introductory level students I mean I this is something that we know students uh entry and introductory courses want to listen to the professor more than their their colleagues in class I also just would acknowledge though that sometimes their classmates annotations were not that insightful they were all struggling with the text Emerson in particular if you've ever tried to teach Emerson uh students really struggle um with some of these texts and so some of the time you know I thought that their annotations were um you know evidence that they weren't they weren't sure about what to make of the text um so this was helpful just to see what they thought they were getting out of the annotations um I then did a lot of analysis of the annotations themselves and this is just a screenshot of the first page of Thoreau's civil disobedience and I did some of what others have done I was looking at what kinds of things they were doing in the annotation and I saw a really wide range some students were doing paraphrasing just trying to do some first level processing of the text I think he's trying to say this I think he means that some of them were doing some more rhetorical reading they were talking about the strategy or why he might be saying that or how that might advance an argument some of them were making connections to other texts um some were asking questions some of them just voiced I don't understand what he's saying here is he saying this or that it certainly was helpful to me before I walked into a class to work on the text to know where students were and also to know which students were at different places um one of the things that I thought was really interesting about the students annotations was that um it it was really clear to students that other students were having um having a difficult time and I think that that did lower barriers to when we got to class discussion that they they knew other students didn't totally get it and it was safer space um but like others the students were not naturally responding to one another the dialogue in the margins wasn't happening students were piling on and each offering their own uh comments on a particular passage but they rarely were um debating or or arguing back and forth um let's see I'm just go a little bit farther I've got one more here um the uh there were things about the way I structured this project that I would do differently so I'm preparing to do um another study I just got IRB approval in my first year seminar class here at southwestern and the first year seminar course here like at many small colleges is designed to do a bunch of different things it's not specifically a writing course it is an introductory introduction to college level reading writing um discussion and other skills and so I am planning to uh frame their annotation assignments differently uh in this upcoming study and to also really push a little harder on the social dimension I think I'm going to have students take different roles I think we'll do different layers of annotations some pre-annotation and then some subsequent annotation um I'm also probably going to have students uh take the role of um being sort of um a pre pre-annotator so posing some expert questions before other students get uh started with text and um and I'm also interested in studying that transfer to the writing so the article that I wrote for pedagogy I don't discuss that very much but that was something that worked I found that students um essays were were starting from their annotations and growing from there and that the final essays incorporated a lot of the observations they made in their annotation so I really felt like that piece of it was a success and in this other course I'll be uh studying I'm going to be um trying to document that a little bit more so I can write about that but the the role that annotations can play in helping students develop evidence-based arguments and and to develop their own ideas about their arguments I think is promising and I want to look at that a little further okay I'll stop there thank you Julie um I want to thank everybody uh on the on the panel roundtable here uh for sharing their annotation stories and give a little bit of time for us to have a discussion amongst ourselves um and also see if there are any questions from the uh folks that are tuned in here um for the panelists or stories that they want to share so I'm going to open it up for our conversation hey uh Jeremy and this is Nate um I've been sort of collecting questions as we've gone along that have come up in the chat um I do notice however that we've already gone a full hour it's been so interesting so far and I don't know if some if any of the panelists do need to leave we should probably let them do that um but we do have quite a few questions queued up for discussion we won't go more than 10 minutes here if people didn't stick around one of the first questions from um way early on uh was something that Dan asked around um around differences about how one might work with beginning annotators like early undergrads or even k-12 I suppose versus grad students um and I maybe that what kind of came around when Michelle was speaking although maybe everyone has some thoughts about that I'll start and then someone else can jump in um my understanding is that the the movements that undergrads need to make from one text to another are a little bit smaller so um Noel I know one of your collaborative collaborators was really interested in how students were synthesizing across texts right but when we're asking early undergrads to do that we're talking about typically a smaller number of texts that we're asking them to bring together um and the movements that they're making are again a little bit smaller so it might be to the in the next class discussion that they're bringing those annotations across texts together or um you know Julie was talking about the papers her students are writing for literary analysis the scale of that work is a lot smaller um when I was working with grad students they're thinking about trying to I've shared before um a metaphor of this moving train they're trying to jump onto this train of discourse that is barreling along and they're just trying to like catch up to it um and things are moving quickly um in that conversation and there's so much more that they're trying to kind of bring together and so that for me is one of the major differences between the work that undergrads need to do and the work that grad students need to do um but but at both levels there's still the the work of do I understand what this person is saying here and how am I making sense of what they're arguing that's work that that's happening across um both levels um what do I notice about the strategies the the author is using to make this argument or to develop this this text um those things I see in common across both levels anybody else want to respond to this I was also struck by the diversity of ways of reading you know we're using this one term reading to talk about a lot of different things in different educational contexts yeah I I just wanted to make a quick comment I had expectation my expectations um were were not at all aligned with with what I saw my students doing and quite a few of my students are would be categorized as developmental writers um and I thought that they they would be responding more to one another and um I thought um as we move further along in the semester and did more work with rhetorical analysis that I would see more of that in their hypothesis responses but what I saw a lot of um and and Julie spoke to this was a lot of really grappling um to to make sure that they understood what the writers were were putting forth so I did see a lot of paraphrasing um and sometimes plagiarism that that turned into some paraphrasing and I and I realized how valuable that it actually was um for them um and I I I really saw that this was something that was that was helping them to learn the distinction between paraphrasing and plagiarizing and and um I don't think they were paying attention whatsoever to one another's work but um if they did they certainly would see that everyone was struggling and sort of doing with that same move um and it was it was very enlightening for for me to see them doing that at that stage um and yeah so I think I think that there's great value in that I just want to add to that know I was talking to a professor at Cal State Channel Islands a couple weeks ago and her assignment specifically was you know trying to get at that distinction that you're saying she'd seen students sort of plagiarizing in their summarizing or talking about the primary literature they were studying so her hypothesis assignment was specifically grab a paragraph and paraphrase it show me your paraphrasing skills just that because that is an important thing to learn it's a stage at some students at red and it's part of the reading process and there's other parts too maybe for later in the semester for further courses but that's a that's a key skill and yeah yeah one one other comment on that back to that question I think um I I now use hypothesis in all my classes and the class I taught last my students at one of my students actually said I would read my fellow students comments more if there was more guidance about what kinds of comments they should be making and I I've usually sort of opened it up like you could do this or you could do that or um but I think her sense was that some students comments were not that helpful because they were just you know they were just reacting and I had wanted to create space for students to have an emotional reaction to the text if they wanted to if that was what was particularly this uh this current class I'm talking about is not a literature class it's about a social issues and the social issues and students do have um personal stakes in those issues but now I'm thinking that maybe giving them more practice with different types of annotations so sometimes they're looking things up and sometimes they're arguing back and sometimes they're commenting on their own feelings or ideas and you know maybe even using Michelle's sort of frameworks for different modes of engaging with the text but that my my students thought was that they would engage more with others annotations if others were getting a little bit more support around what kinds of annotations might be helpful and so I've been thinking about that a little bit and again I'm dealing with first-year students here so there are students who this is their first actual college course and um they're hungry for guidance I would imagine the doctoral student situation is obviously totally different. Nate anything else you want to surface from the chat? Well yeah um there's a couple of other questions actually and again I don't want to keep people past their comfort zone but um uh Anya uh brought up a question around um good annotation exercises to get students started and we did have a quite a bit of discussion in the chat around annotating syllabi and Dan brought up the idea of having students annotate rubrics so there's some ideas already circulating in chat but maybe if any of the other panelists have some ideas around um how to get students sort of warmed up and started with annotation? I'll make one really quick comment um the the very first text might the very first time used hypothesis the very first day of class I signed them to set up hypothesis and then start annotating Benjamin Franklin and um they just did it I mean that's not to say that there isn't value to giving them warm-ups but that was part of what was surprising to me about using hypothesis is that within 24 hours I had students annotating a text with very little um there seemed to be very little barrier to them to jumping in and and I found that sort of surprising because some of the research I'd read talked about the technology itself being an issue so I'd be curious to other thoughts about that but um I kind of just threw them in and um and they got started and it worked fairly well I do like the idea of having them annotate like rubrics and even the course website that could be really interesting I I did not do this but as I as I heard the question and I started thinking a little bit more about it um I have in the past had students as a group um analyze images painting um uh non sort of nonverbal do some nonverbal analysis and I'm thinking about it you you might be able to take a poem um and and have students sort of try to analyze it and and respond to it that might be an icebreaker where everyone's definitely starting off on an even playing field where they're probably used to being invited to interpret um pieces of of work like like poetry in a way that they may not feel comfortable interpreting or analyzing um a more academic text so that's just something that I just thought of off the top of my head that might be a way to kind of get them get them loosened up and to feel like the that it's it's safe to have some guesses and some some interpretation I like that a lot Noel also Ryan Rich at uh I think he's at Buffalo now um has done a workshop where he does really analog um uh annotation practices where he'll print out a blown up image of something from the news or a paragraph and then have students gather around and kind of put post-it notes on it so just reminding folks that this comes from a you know traditional for some of us at least you know traditional practices that we're familiar with when I started using Genius I would I would put the roster up and have students annotate and just highlight their name and like three questions like put your favorite you know put a picture yourself and your favorite uh I don't know something a little goofy get into know you kind of thing just to get to know the tool but also to introduce each other um introduce themselves to each other you know maybe we have time for one more question um although obviously we could stick I'm sure Jeremy and I can stick around for anybody who wants to keep talking but um Jennifer brought up the interesting idea of if you guys have seen any differences or have any thoughts about how annotation might be different in a face-to-face course versus an online or hybrid course I have only used it and and fully face-to-face courses but Noel you said you used it in a blended course well um so my course was blended in the sense that um there's actually another term for it and it isn't blended because blended is hybrid but um so half of my class um what were made up of of uh students who who had been placed in remedial English non-credit bearing English and half of the students were students who had been um placed in four credits so it was half developmental and half um sort of traditional one-on-one so that's what it this I can't remember the word for it but no so I I've only taught face-to-face and I think I would be um I think I think I don't know about using it for first-year students um in a in a hybrid or an online setting I mean I think it would be a great tool but I don't know that that you would you would always um get the kinds of responses maybe that that would allow them to um sort of really express themselves I think students who who've been student college students for longer maybe upper level courses I think that might be a better way to to to use it in a hybrid setting but for the first year I think that might be difficult one thing I think you'd you'd notice more in the in a fully online section is that it's not it's not just the annotation and the close reading and all the all the elements of the reading process but the the social is going to become more important right because that that could be the form in which students are predominantly interacting I think it's a deeply I mean for me it's the closest I've ever been to a seminar class is you know being on the same page as folks uh and annotating and you might see a lot more of that social behavior coming out uh certainly would be a good time to encourage it um any last comments from our panelists on that issue most folks for teaching face-to-face I don't know if we have a lot of the online experience in this context um I want to thank folks for sticking around um as I said this is the first of a series of webinars on uh on research and annotation and hypothesis hosts webinars on annotation education throughout the year Nate and I also really aspire to to cultivate a community of practitioners around annotation and education I believe I'm all confused by my windows here but I believe I'm sharing the contact information of our panelists here if you want to follow up about their research or connecting and possibly the collaborate Noel and others are working in a research group getting together for panel presentations hypothesis hosts events we have an annual conference coming up just a couple weeks in DC if you're in the area should definitely try to get Chris to come down Noel I'd like her to come over from uh from Virginia that'd be great um and we host conferences piggybacking on various ed tech and scholarly conferences throughout the year to try to really cultivate a community of practitioners around annotation and hopefully some of you that are presenting but also some of you that are uh participating attendees here might join us face-to-face sometime or just continue to stay tuned to the to the work that the community is doing so thanks everybody for participating and especially the panelists for sharing their work