 Greetings. Greetings, I annotate. It's really lovely to have folks here. We are entering into the final, the fifth educator office hours in what has been a week long of I annotate. I anticipate that people will be trickling into our session over the next few minutes. I'll just briefly say as by way of introduction, my name is Raimi Collier. I'm an assistant professor of learning design and technology at the University of Colorado in Denver. It's been a pleasure over this past academic year to get to know everybody who's joined us today for this particular set of office hours. And so as we get started, just a reminder for folks, we have a chat that people can use as they, you know, want to say hi, tell us where they're maybe joining from, tell us where they may be teaching or what they may be doing. We can also use that chat for questions that we can then upvote and bring forward into the session. And we'll see who joins us for this informal conversation this afternoon. That's all I'll mention by way of at least introducing myself and the platform. I'm happy to answer, again, logistical questions as they might arise. But let's just say hi to who we have here. Tell us a little bit about who you are, where you're joining us from, what you're doing. And maybe we'll begin with Laura because it's always nice to pick on Laura in these circumstances. Oh, thank you. Thank you for having us. We're excited to be here. I'm speaking for all of us, even though I can really only technically speak for myself. But I'm Laura Roche. I'm a PhD candidate at Indiana University. And I got involved with this project with our hypothesis work over a year ago now just by way of curriculum design and bringing it into our college composition classroom. And I've just been learning about it, about the platform and about social annotation ever since. On to you, Mary Helen. Thanks, Laura. So hi everyone, I'm Mary Helen Truglia. I'm also a PhD candidate at Indiana University. But while Laura and Sarah are in the rhetoric and composition side, I'm on the literature side. And I got involved in the project after using hypothesis and several of the literature classes that I've taught. And we're really excited to be expanding it across our whole first year composition program and some of our lower level literature classes. And we're excited to discuss annotation and specifically the work that we've done with you all today. And I'll pass things over to Sarah. Hello, I'm Sarah. As Mary Helen said, I am in the rhetoric and composition part of IU's English department. I got involved in this project a little under a year ago, just as I was becoming more increasingly interested in like writing studies, writing program administration, and stuff like that. So that leaves me to pass it to Justin. Hi everyone. My name is Justin Hudson and I'm an associate professor in the Department of English at Indiana University where I specialize in digital rhetoric and technology and learning among other sort of hats that I get to wear. One of the things that I sort of spearhead is our online 131 curriculum, which is our freshman composition course. And I think this project really began for us last, well, I guess it's been right before the pandemic hit. We were asked to redesign our online course and make some really strategic changes. And one of the cool things we decided to do was to include hypothesis to improve our peer to peer engagement among the students in our classes. And at the time, we were only looking at like eight sections of an online course that coming fall. But then, you know, pandemic, you know, university political decisions and we decided to just go with our online course design for all of our freshman composition program, at least the regular 131 courses, which ended up being about 60 sections a semester. And so our hypothesis decision early on for a small really boutique, wonderful engaged peer to peer dynamic became sort of a staple now of our first year writing curriculum. And so we decided that we have this giant thing now facing us at scale. So let's collect some data as we were going to anyways for programmatic review, but let's turn it into a research engagement and see what we can learn from what we're doing, how our delivery works, how our design works, how the students are using the tool, and so on. And then from that we stumbled into Raimi through hypothesis, and I've known Jeremy at hypothesis for quite some time. And once we got hooked up with Raimi and started thinking about the strategic ways we could operate at scale and do research at scale, we've opened up a lot of really fun avenues for how we are thinking about hypothesis, not just as writing teachers and literature teachers, but really the sort of learning principles and educational practices that go along with either the good decisions we've made or the decisions we know we may want to change coming next year as we manipulate the tool a little bit in our practices a little bit. We're happy to be here and happy to answer questions and share with you what we know or don't know at this particular juncture and update you on things to stay as they unfold. That's brilliant. Well, I just again, I want to welcome Justin and Laura and Mary Helen and Sarah. I'm just noticing that our session is almost doubled in size just over the course of our introductions. And so I'm just going to repeat for folks who've joined us in the last few moments, a bit of context and another invitation for people to say hi in the chat. Maybe tell us where you're joining from. Maybe tell us why you're joining this particular educator office hours. If you've joined this session, this is our final, again, educator office hours of the IANA conference. These are intended to be very informal and laid back conversations, primarily pedagogically focused, not diving too deep into the technical details of any particular tool or even hypothesis. What's unique perhaps about this office hours is that we have of course, not only a whole mix of wonderful educators, but also people who are involved in some research. And so, again, as Justin was just providing a bit of context and we can also drop a link into the chat in a few moments, we have been researching this particular project at IU over the past year and will continue to do so. Everybody in this session, Justin, Laura, Mary, Helen, Sarah, we're all wearing multiple hats, hats as educators, hats as researchers. And just to kind of kick off the conversation and then see where things go. I just be really curious to hear from from all of you in, you know, whatever order makes sense and whatever ideas make sense. We've been engaged in this project for, you know, a year now of designing this research of facilitating these classes of collecting this initial data, but you've all seen in a very kind of hands on way in a very kind of close way, how social annotation has been supporting students in their learning and again the particular context of English and composition, etc. But also how you see educators making use of social annotation. And I'm just be curious like what are some of your like real strong takeaways right now or insights or even questions that you really have at the front of your mind, given this past year of work that we've all done together. Yeah, we actually had this question on our Wednesday meeting. So our regular which I missed because if you had been there, we would have been able to inform you about writing this. It was a rehearsal for this. It turns out we don't know what we want to ask right like it's not and I say it in this way we when we began this project and and designed it around including the tool around peer to peer engagement. So it was really a way to try and maximize the experience of online learning, like we're trying to improve that for students, and then we went to scale we really know we need to do things that we're going to be different than the traditional every lectures in a zoom box approach. But as our course is designed in such a way that students have to read five, the 131 course anyways students read five course essays, and they're asked to work primarily analytically with them, and then apply them in their writing and their thinking throughout the course. And, you know, one of the things that became least from my perspective that became really important was, okay, well if we are going to include something like hypothesis or social annotation activity, which I think is a wonderful learning tool. Right. Is it going to make a measurable impact on their ability to use those texts and to leverage those texts and their writing, you know, because again as a writing teacher comes back to how can I help the students improve their writing and I always say there's no substitute for clarity of thought so the better they understand their thinking with the text the better they're going to write with the text. And so that's where my the sort of the position or the impetus began with thinking about this as a research project which is, in what ways can we determine or can we even determine if social annotation improves students understanding the text, and or if it actually can be tracked to translate into their writing in the course. And so I think that's what one of our primary questions where we began. And then yesterday we started getting or Wednesday we started into all kinds of now that we know so much more what would be nice to ask like can we get hot spots like a map of all the text and see where the quotes are showing up. Those kinds of weird or wonderful things so but I'll let the other speakers I know that we had some really great conversations on Wednesday about where we go next now that we have our first sort of large chunk of data in place to use. Sarah and I have talked a lot about our interest in the way students collaborate in the annotations right. We've worked, we've been toying around the language of collaborative annotations versus social annotations. And we've been thinking a lot about how students respond to each other, and take those responses into their writing assignments. So that the reading and writing processes aren't so separate. It's not that they read this text and then they write about it. Three weeks later, it's that they read this text they're thinking about this text they're talking about this text, and it naturally progresses into the written form. And so our, a lot of the questions that we've been asking and we've been texting about during the meetings have to do with the student to student interactions. But obviously there are so many different ways for us to engage with this data because we've got information about instructors, we've got information about instructor student relationships. But it, like, like I think Justin was alluding to, there's a lot, we've got a lot of data, we have a lot of students with a lot of annotations. So trying to decide where we want to focus our energy has been an exciting challenge, especially this week as we've been having these conversations more and more. Something else that Laura brought up on Wednesday in our meeting that was really interesting to think about and it was things that were sort of like in the back of my mind. When we think of like research at least like in our experiences with like rhetoric and composition it's more smaller like focused like case studies here's what I did in my classroom but here we're working with this data of so many different sections and so many different students that it almost feels like overwhelming with how to even know what exactly do we want to ask because there's just so much of it. Yeah, another thing that we've been kind of thinking about and wanting to work with in terms of the data is also the students annotations and how that might show up later in their writing. So we're looking not only at their direct annotations on the text and the ways that they're as Lauren Sarah we're saying speaking with each other and thinking about interacting in a more kind of immediate context with the text but also the ways that reflects later in their writings in the essays that they're asked to write across the course of the semester. So, annotation they might make especially towards the beginning of the semester can then become relevant as they start bringing together all of those analytical ideas and move towards later texts so we're really grateful to have this access in the research project to not only the annotations themselves but also to student perceptions of their annotations to instructor perceptions we've been working with another colleague who's not joining us in this session today but who's in the Department of Education he's been actually interacting with instructors directly to kind of study the ways that they've been building their courses around this and the ways they've been asking students questions. And then we'll also have access as part of the study to students written products at the end. So that's one thing that I'm particularly interested in seeing the ways that their annotations and the ways they've been able to interact with each other showing up in those final analytical essays we'd be happy to talk about that as well. Yeah, so I really appreciate that I just also appreciate that you're sharing such, you know, very perspectives on this really broad project and I think it shows just all the different points of entry into the work which is which is really exciting. I'm looking at the chat a little bit and and I'm also I there is one questions come through but I did want to kind of scroll back in the chat. And I see that we were joined by by Candice Candice welcome. In your introduction you mentioned that you're redesigning capstone courses that your particular institution and I think that at the end of the day, this project and again all the expertise that we have here is ultimately about kind of course design very intentional course design thinking about supporting faculty, how to do professional learning for instructors, how to get, you know, new kinds of activities into the curricula, and how to support students and engaging that over time and I'm wondering if, again, any of our of our guests today, maybe have some perspective, maybe for someone like Candice, who is in the process of redesigning courses. What you've all gleaned about course design, and then support for instructors, when something like social annotation and specifically hypothesis in this case is a really pretty central component to those redesign activities. Like what does that look like. Well again what insights might you be able to share with us. Yeah, so first I let's make sure we don't forget to get back to the question in the chat or the Q&A at some point. But I think, you know, for us it begins with, you know, what are what are our course design goals. So, as a long time, you know, online educator and course designer and whatnot it's always a matter of for me I focus on what kind of learning experience can we create for students I think typically speaking we design courses and humanities, especially in English. We design courses around a set of readings and then figure out what we want them to do with them, which is sort of backwards in terms of, you know, actually how a lot of course design works, which is that we want to focus on one of the learning experiences that we need to, or that we can use to accomplish our goals or objectives and outcomes. And then in my case how can we create, perhaps better more engaging more dynamic learning experiences whether it's through active learning technologies, whether it's through, you know, specific kinds of assignments that make them, I do a lot of digital literacy stuff so having a student make a video instead of writing a paper is a fun way or a introducing a different mode can ask them to create and engage with the content differently so those all kind of are like floating when we begin to think about these kind of courses courses a capstone course typically though is a like a seminal major of some sort of achievement or some sort of engagement and for my experience as a teacher and what I've seen at least in some of the early conversations with our 131 online design is that the role of hypothesis in this dynamic, particularly reading text and reading critically has been perhaps second to none like it's amazing how much of a difference the conversation is or how it operates simply by having students have read and annotated before coming to class. Even if they didn't read the whole text, if they've taken a minute or if they didn't annotate on they just read the college their peers annotations, like the conversation starts about three moves further along. And then it went before it took me a while to adjust to that. And I think, you know, going forward, we need to an hour and we're going to, I think make some changes to our design, such that our instructors are better prepared for what that looks like in the day to day activity of a class not just that pre class stuff but so I think for me it begins with that like can I use this in this case in the hypothesis specific conversation. Can I use social annotation to improve the quality depth, or overall framework of engagement the students are going to have in my course and is that going to give me a better learning experience. Can I craft a better learning experience using this tool. And then how can I build out of that. So it's not like it's simple like well this is an easy fix it's more like this is a tool but it does enable a whole a new set of pedagogical considerations and strategies. In the same way that if I have students work and make a podcast that also enables a different kind of set of strategy so I think to me that's a tension that exists no matter how we approach the fourth design conversation that just, but it's a great place to start because what if I had to pick five experiences for students what would they be or three and build out of those and sort of work backwards from them. That's been successful for us with our online model anyways. Sort of extending that. I've been thinking about how an instructor's use of or implementation of something like hypothesis represents their pedagogical values. So if you just assign it if it's just that you have to do this, you need to complete five annotations come to class be ready to talk about it. It's a very different use from the instructor who says that you need to respond deliberately to three of your peers, you need to ask a question of the text you need to ask a question of appears annotation and through that sort of guidance. Because if instructors have an opportunity to articulate what sort of values they have for their classroom. It's an opportunity to foster not only relationships but sort of build empathy between students, consider different teaching teach them how to engage with perspectives. And there is of course this performativity to the rotation the social annotations right students know that it's not that they just have to cross off the annotations from their to do list it's that they have an audience beyond their instructor when they're asking questions of the text when they're responding to a specific quote for example. And so I think when it comes to course design. It's not enough to simply include a tool, we have to think about how we use the tool and how we want our students to use it. So just sort of extending Justin's point there. Yeah. Yeah, and that was one thing that really became evident for me, moving from doing this curriculum and previous iterations of curriculum especially in person we always required annotation like this that has always been a part of the class. But I think one of the great boons of social annotation particularly as Laura and Justin were talking about is that it engages students with each other and shows them that they really are a part of the conversation, which in either a composition class or an entry level literature class is something that's very difficult to get across to students I think in those early things especially if they're not traditional humanities majors or if this is not a kind of reading technique that they're familiar with. They may feel very alienated from the text and not, and especially if they have questions or they don't feel like they understand a particular portion of it. Social annotation is a really nice way to give them that access and to provide them with kind of a low barrier to get across to start asking questions about the text to see that other students are also maybe engaging with the same portions of the text. And as Laura was mentioning this also is such a great thing for the instructor and if the students are able to do this maybe before they come into class to read that text, the instructor has access to those annotations and then is able to dive into that discussion with such panache because they have been able to see where the students are confused or where the students are really excited and kind of start there or maybe see on the opposite end sections of this text that have been left completely unannotated and maybe start there to say, Hmm, I wonder why the students are leaving this alone. Was it because they were confused? Was it because this is a less interesting section of the text? Maybe it needs to be explained in class. And so as opposed to replacing class discussion, I think one of the really important things about social annotation is that it comes in as a compliment to it. And that's something that we've seen across both in my own personal teaching as we've moved into this new version of the curriculum. And I think it's some of the data that we started to see from the Actually, Mary Helen, you started to touch on one of the things that Jeremy asked earlier, I think, which is some of the differences between like the writing studies perspective of freshman composition, which includes reading, but a very specific targeted kind of reading towards a writing target outcome versus the literature courses, which you spend more time in the L to our L to three, four and fives, which are introduction to what poetry, prose and fiction, right? Those are the three we work through. Yeah, poetry, drama and fiction. Yeah, which drama and fiction. Sorry. And then we also have a prose course, but it's not in that nonfiction prose, which right now we are studying, but maybe next semester, maybe next time. Yeah. But it does. I think it raises questions about how this something like this as a decision operates across what is a clearly related field but also not a related field right in terms of some of the pedagogical things that we do. Yeah, Jeremy, for folks who are following along here, Jeremy had asked a question quite a few recently, but one is in the Q&A section of our chat about any differences, perhaps between, again, the use of social annotation as a teaching tool and rhetoric, again, compared to English and you know, and literature. And I don't know if folks have other thoughts about that that we might chime in on before we move to any of these other questions. I mean, it's hard for us because we don't, I think Mary Helen's the only one who legitimately is occupied both positions, teaching the literature course and teaching a writing course. So, you know, Mary Helen's probably has the most insight. Yeah, I think I think the main difference there. I mean, they're both, as I was saying, I think it's a valuable tool for either one. As Justin mentioned, I think the pedagogical concerns are a bit different. In the composition classes, one of the things that we're trying to work them through at the heart of the class is really having them understand what good analysis looks like, how to explain their ideas, how to integrate sources and through the ways that they can translate that into lots of different contexts. So that's one of the goals of the classes, not just that they're like, well, I took an English class and now I know how to write for English, but rather that the kinds of heuristics that we do in the composition course are translatable across all sorts of different disciplines because they're using techniques like using a source as a lens, like thinking about comparing contrasting different viewpoints and then being able to explain what's going on between them. So the difference between that and kind of the things that we're focusing on in composition course and then those entry-level literature courses is that we're really thinking about the ways that each one of those genres functions. And so in each literature course, we're really asking students to kind of build on, they will have hopefully, at least if they're going through the university program as they're supposed to, have already taken that first year composition course prior to taking the literature courses. And so it builds on those skills as a way to really get into close reading techniques in each one of those genres, whether it's poetry, drama or fiction. And then to be able to talk about those as ways of seeing the world, as ways of understanding what each literary genre can do and the processes that we move through. So things like in the drama section that I taught about a year ago now, time just has flown by in the past year, I'm sure as it has for everyone. One of the things that we're able to do is to do a comparison between Shakespeare's original text of him at summer night stream and then a kind of modern updated translation. And so I asked them to do a hypothesis assignment on each one and to look at kind of similarities and differences across what's going on there to think about the way that an early modern text can be translated into something so much more modern, but it retains some of those same themes. I'm still really looking for kind of some of those higher order concerns, because hopefully they already have that foundation of the excellent work that we try to ask them to do in that composition class. So I'm sure I'm kind of just talking around the question now so I welcome you guys to jump in and I knew a bunch of other questions in the chat. That's awesome. No, that's fantastic. I'm sure that's really awesome. Really good. Real quick. So I'm going to touch on Chris's question real quick in the Q&A and then because I want to have Laura talk about the social annotation and empathy and what she meant by that because I think that's a really fascinating point, but because I can answer Chris's question relatively quickly in that we haven't at this point spent a lot of time thinking about the historical canon as a framework for how we've been approaching the social annotation activity. It's one of the sort of tensions of being a writing studies framework of a program that's connected to rhetoric that is informed by our research is really more about because of the course design about analytical writing, but I think it's fascinating to think about how some of those canon elements absolutely are operating and my first response is going to be like I think the primary mode of this tool would be Inventio which you then later point Chris I think in the chat and because it is by design interpretation is also closely related to invention heuristics versus sheuretics and how those terms operate in our canon. So we haven't done a lot with it in terms of memory and the way you sort of operate about exporting comments and making them available in that regard, but it is an area I think that we really haven't spending time thinking about it yet. So, which is odd given that you know, in theory, I am a I am a rhetorician right so as a scholar in this field you think I would have been better at maybe paying attention to those things. But yeah so I think within the canon though, Inventio is obviously connected to it. I think memory could be depending on how you want approach it the text is revisitable so there's a kind of tension there, and then we have issues with like delivery and arrangement and style but the style one is also fascinating for me like I'm intrigued by how many times students include links or images in their comments rather than just text. And so that's not only it's a content choice but it's a stylistic choice. I mean how many I want more tiktoks in the comments is that is that so much to ask I don't think that's a problem right so so that's my own take on that I'm happy to have an extensive conversation about where and what that would look like but I'm I too am fascinated by this idea that Lars brought a couple times in our meetings and here which is the relationship of the opportunity to be empathetic. That's one another and the relationship between social annotation and empathy. So Laura I want to know more what do you mean and how does this work. What I think people should know is that Justin is my dissertation chair. So the way he's asking this question is a form of pushing me to define my terms. And what I will say is I don't know that I'm prepared to define my terms yet but I do think that my interest in social annotation in encouraging students to sit with the text and see it as a living breathing thing that they engage with along with other people is really important because there's this caricature of the academic as an isolated sort of like loner and when we perpetuate that image when we say that you sit with and you read alone and you write alone and we only read and write alone. We. I think we do our students a disservice and so when I think about social annotation as a tool for developing empathy I think about it. In that way because I think empathy happens through exposure. And as you're reading through students other students your peers for commentary when you're reading their questions you're being sort of forced to consider the fact that other people are engaging with the text differently just by nature of being different from you. And so I don't know if I'm really answering the question I don't know that I necessarily would know how we identify when a student is being empathetic in their comment. This is something you know Rami we chat a little bit about yesterday in our data meeting which is you know involving sentiment analysis at scale to not only maybe to determine what's going on at least with some sort of sentimental kind of tech language but specifically to maybe mark a subset of of annotations that we would then do a more. So we use data science basically to identify a subset to do a qualitative look at our rhetorical analysis of to see what is going on in those exchanges. I don't know. Well I'll just mention a few quick things and this is to Curtis's question earlier Curtis welcome it's really good to have you here. I think that we can we can put to some examples. Let's say of research where qualitative to your to your observation Curtis qualitative analysis of smaller sets of annotations do show respondents whether those are students or those are educators in some cases doing things like showing affinity. Kind of sharing personal experience being responsive to vulnerability sharing kind of critical introspective commentary. We might think of those as maybe aspects of empathy and looking at how those annotation exchanges provide some evidence of kind of more empathetic interaction as mediated by social annotation. And there's a lot of research I shouldn't say a lot but there are quite a few studies that also show how students perceive their learning with social annotations. That's a step removed from the annotation content. But again how students perceive their use of this type of again social and collaborative experience. And they do perceive it as strengthening their ability to connect to peers or appreciating diverse perspectives. Or again learning as was mentioned you know here just from the fact that they can have access to their peers thinking. So again there may be aspects of this that we might broadly construe as kind of empathetic reading or to Jeremy's point here and I was just trying to find some related commentary on Twitter but Jeremy mentioned this idea of annotations a kind of listening. And one of our K-12 colleagues who was presenting earlier this week Courtney Cluthman she and I had an exchange on Twitter a little while ago about annotation as very much related to listening. So I'll try and pull that up and throw that into the mix here but anyways I appreciate this line of thinking it really resonates. So the one of the chat responses is from Jeremy that if they're being empathetic that definitely ought to be looked at for or for in the response of those to whom they're being empathetic towards the question. Yeah how would we then also track like audience reception of empathy in a flattened you know it's at the point it's text so it's an artifact this is strangely removed from the person even though it is never removed from the person so. One of the things we had brought up a couple times in the practices and things we thought we might be seeing as if you know if I'm over we have students have to make so many responses and are there certain types of annotations that become viral in terms of like they've made a threshold of like engagement from other students and that seems like an inferior seems like a great idea except that if it's a group of students who only ever respond to each other's posts and then you know it's a sub community and so then that changes the dynamic in the same way that empathy may or may not be impacted by those friendships those you know those kinds of things so there's so many layers to how we would even begin to track or identify this even within one course right let them least of all what across 1300 participants I think this will what our final numbers are at this point. And so yeah I don't know that this is these are fascinating questions that's that's just what I'm saying like I don't know what to do with them like I will get there right right I mean we'll get there. Because you mentioned sentiment there has not been at least in my reading of the literature, many if few empirical studies that apply sentiment analysis to a large corpus of annotation data. And I'm going to kind of leave the jargon there we know we did one study in the marginal syllabus project that actually showed that the deeper a thread grows right so again exchanges among participants over time. The more neutral a sentiment actually became, which is hard to interpret against sentiment analysis depending on its specificity. Does it does not highlight certain you know notable patterns so I'll just leave that there but I think there are a lot of ways to get into this question of how you would begin to define that term back to your, you know, back to your point Laura, and then look for you know evidence of it and then try and make claims. Now to Justin's point about what that might mean as students are making meaning of their readings. I think something that I've been struggling with on this team is figuring out where I want to invest my time. Like, do I want to look solely at social annotation of the text in the text, do I want to look at how social annotation then affects subsequent assignments, do I want to look at how teachers are involving themselves in the social annotation process. And I think we alluded to it earlier but really keeps insisting that this is a very unique and exceptional opportunity to work with an enormous data set. And so, trying to like stay open to the possibilities that the breath can allow for while also saying, I think I'm going to dive into this one thing is really really challenging. And I don't know if Sarah and Mary Helen have felt that way. Justin loves to sit in the vastness and just say, who cares? Let's take it all on. We're going to do it all. Who needs sleep? But for those of us who don't want to age more rapidly, we set boundaries, right? So I guess I'm interested in how Sarah and Mary Helen, you're negotiating, rhetorical listening. Yes. I just felt like we couldn't talk about like reading with empathy without bringing in Ratcliffe's ideas about rhetorical listening. I don't know. I read her work somewhat recently as I'm preparing for like comprehensive exams this fall. One thing that really stuck out to me about her work was how she talked about how when engaging in like this type of rhetorical listening, which is different from her for like rhetorical reading, but similar at the same time, is that there's like a necessary two part component or like two components here. Like one is like understanding the other and also like better understanding oneself. And that's one thing that I think a lot about like how does writing help us better understand ourselves. So I wonder if like we see that playing out in these annotations as well, but this is something I'm like just thinking of now. This is not something necessarily that we've talked about in our meetings or anything. Bring it rough to have thinking Sarah, bring it forward. This is what it's all about, right? How's we supposed to get there? Right? That's the question. But it does, it does raise, I think Laura to your point, right? I think it may be helpful for folks to understand like what we're dealing with what we're looking at. It would first off every course has a set of texts they choose from. So we have like a library, a defined library, if you will, of courses there are texts there included in their course to have students read and annotate in this particular. So we have a where we can look across, let's say these five courses have all used this this text. And so we can look at that as a framework. We've also gathered all the student at their own writing for the semester what they've produced, which if you take 1300 students and you gather, you know, what is it, three course assignments that are like 1000 more and each or more. So they're long form essays, right? And we have the social annotation data. We have instructor feedback data. We have, you know, course section breakdowns. We have grade distributions both on assignments and in the course that are available to us. So to Lars point, it is it can be overwhelming to figure out as a researcher what we want to do. But as a as a pedagogue as a teacher as a designer, like the even the without knowing what the data suggests yet, we've already learned so much in terms of, you know, sort of what we don't call the best practices anymore. We call them vet moves, right? Laura, that's what they are. Sort of the the moves that the more experienced teachers have been doing with them, how they're investing themselves in the tool or what their experience has been like. Because they we we've used this tool in our classes for two semesters. Now we're going to use it again at scale next fall and next spring as well. But the first the first semester last fall, we didn't collect any data on it because we were still getting things in place and getting all the parts and trying to figure out how we're going to gather all the stuff. Right. And so what that did is it allowed us to have a semester of like a dry run with instructors, whoever there's a first exposure to hypothesis. And we got a lot of feedback, some good some negative, mostly about course design and mostly, you know, about how the online experience wasn't wasn't working for them. And so we made some changes. But but ultimately, I think that stuff is going to resonate no matter whether we get lost in the nuances of a single course analysis. Or like me, you're like, let's find let's just look at the large scale data and see what the implications are about, you know, just basically descriptive. Like Raymond, which we've been talking about how many students are annotating. So we, you know, we know for a fact that in our W 131 courses, students should complete 25 annotations for the year, five annotations per course essay, three additive, which means, you know, original compositional kind of things and two responses. They're they have questions, they have themes that can do whatever they want the instructor can control that but each student should complete five per essay for a total of 25. And so already we can tell you that not every student hit the 25 annotation mark. But then we had some students who have well over 25 annotations. And so like those kinds of things are already kind of indicative of differences between courses differences between reading and most likely as I think we're finding at least in my initial assumption at this point is differences between instructors. And I think, you know, instructor buying has been maybe one of the single most important components because I think we've been seeing higher numbers. I think from instructors who also participated in our focus group for, you know, advanced engagement. So there's a there's a kind of, you know, just basic correlations that are already occurring. Despite the fact that we have this immense amount of data to try and figure out are their patterns are their things are their correlations are they statistically significant or not. And how we can go about that to report this back so we can all make better decisions about how we employ these tools that are in our classes. So that's that's sort of the framework, whether we get there, I don't know, but That kind of leads into the question I've had in my back pocket for quite a few colleagues. During this this hope of just during this that Diana take commerce I think I asked this during some of our previous educator office hours earlier in the week. And then some other sessions here and there and I'd love to hear from all of you about this. Again, the session attendance keeps growing so maybe this will be useful for folks who are just just joining us even in the last few moments. You know, we've been involved in this project now again for at least a year, some of us longer again depending on connections and history with Justin your whole team. You've all been again wearing multiple hats, and you may also have different kinds of teaching responsibilities coming into this next school year. I just really curious like, given what we know now or what we think we are learning now. How do you see yourself employing social annotation either in classes that you're teaching, or how do you see yourself supporting instructors, who might be using social annotation. What are you saying to them, what are you planning to do. I just emailed Justin about this because I'm teaching a 300 level writing intensive course in the fall and I really would like to use hypothesis. But we don't, unless we get like a grant for this research project, we don't necessarily have the money for the department to pay for hypothesis to be used in my classroom. And so I'm seeing that Candice asked the question of what are you noticing about instructor buy in what are the obstacles for instructor and asking my students to pay for hypothesis for me is really difficult. When I don't have to have my students pay for texts. I don't when I don't have to have them pay for technologies I don't. And so I'm really trying to weigh the pros and cons of having students invest in hypothesis for our time together in the semester, because I have been thoroughly convinced of the benefits of its it as a tool in the classroom. And so I am really hopeful that I'm going to get to use it with my 300 level students. And if I when I do, let's say let's manifest it when I do. I, I am really hoping to encourage add the additive comments that Justin has referenced, I really hoping that the students will ask questions of the text and respond to one another's questions before coming into class, because it's those before conversations, it's being able to reference and point to something that Mary Helen said in the course text two weeks ago that gets students excited that they were heard in some capacity before they even came and stepped foot into the classroom. So that's my plan. I don't know. Well, let me add real quick, I want to hear what Sarah and Mary Helen have to say, but to Lars, to your point, though, one of the purposes of this kind of research is to help us make the case who the department and the college and the university like this is not just another you know it's a tool we have a lot of tools we have a lot of enterprise level agreements that we pay for, and students pay for things all the time so there is, that's, you know, this is it's no more expensive than another textbook if you think about in those regards for students, but our part of my goal is to demonstrate why this tool in fact is does have measurable value and measurable impact and why every college should invest in it, right, for every course who wants to take advantage of it. In the same way that we, we know we pay for things like the Adobe licenses the Microsoft licenses that are tools that they just think are commonly acceptable kind of things right. So like that's, that's an ongoing purpose of behind what we're doing not just because I think it's a great learning tool, but because I think it's a learning tool that needs to be invested in at the university level, such that it has a, I think its impact can be more measured by the way but so, and then we can work out the other stuff behind the scenes. But Sarah and Mary, Mary Helen what are you what are you been telling people. Yeah, I think one of the things to kind of address Candice's question is one thing that we've been seeing is especially newer instructors have been a little like it on one side of the spectrum either either really engaged with this tool and think that it's really fantastic or very wary of it and don't really know how to use it and just think that it's another I think one of the challenges that we've seen instructors come against is that they think falsely. We think that it's just another thing to grade like it's just another kind of evaluative tool to use and so one of the things that we're hoping the study will be able to show them is both through getting to use it in practice and also through seeing what we find in the study that having this extra access into the text this additional access not and again I think that's one of the things we're trying to show them is it's not a replacement for students doing the reading on their own for students taking notes for having class discussion like it's not meant to supersede any of those but rather to really be a part of the entire curriculum wherein the social annotation brings another level of engagement with the text. And so I think one of the challenges that instructors may have faced in these kind of early stages especially if they haven't used it before is that it felt like just another requirement hurdle for them to do like I like Justin was saying we've student is asked to make five annotations on each of the five texts throughout the semester and so then instructors like well now do I have to go in and you know count and see like ah yes Bob made five annotations Bob gets five points and that's really not the not that we shouldn't you know also give them credit for doing work but that's not the main impetus behind using the tool the main impetus is to really get them to be able to interact with each other interact with the text. And so exactly kind of as Jeremy just mentioned in the chat one of the things that we're hoping to show them is that this deeper engagement with the text and these multiple level conversations they're able to have in addition to class discussion with social annotation makes them kind of more careful readers and writers which will result in we hope and this is another thing that we're at the beginning stages of the study so one of the hypotheses haha that we are hoping to find is that this really does improve kind of the quality of their writing because they're spending that time engaging with it. And I'm thinking about hypothesis much more from like a student perspective than a teacher perspective because I've never taught with hypothesis before because I was teaching public speaking last year and that doesn't really have text that you read but thinking of this from like more like I already use hypothesis as like a student like even in a grad class or something. It's much more inviting to respond to people's comments than it is on like a canvas discussion board or like a blackboard discussion post where sometimes teachers will say or instructors will say OK post something and then respond to two other peers and then there's like jokes about how your posts are like great job this was so interesting and they put like a word count on how many words you need to say but something about like annotations on hypothesis is much more specific and you can dive into specific parts of the text and it's it's less like you're sort of like floating and you're like I'm going to talk about whatever I want it's pinned down to something much more specific so I think it's more inviting in that way. So that really echoes a comment that was made by one of our k-12 colleagues in a panel earlier this week Morgan Jackson. She emphasized just time and again this kind of proximal relationship to the text and how we're just continually kind of inviting students to return to that close reading and then that kind of close and things can then build out from there but to your point here they don't kind of start in the kind of stratosphere right they're like right there in that very proximal relationship to their reading. Anyways that's that's just very much on my mind and really echoes what you're saying. Yeah. No, well you know this is the two couple things that have come up for me that I'm I'm I continue to struggle with it right and one of which is that as we have currently designed it in the online W 131 course. The annotations themselves aren't meant to be graded in terms of quality they're just meant to be like you've completed the activity and it's and it's meant to be a quick read there's been some some confusion on on that which is probably part of the tension there but it's been like, you know you do five and it's because it's through our canvas LMS the speed grader pulls it out very quickly you can identify how many things so if it's accounting system it's pretty simple but it's not make a proxy for participation just yes it really counts as part of our participation right so. And so there so there's that and then we have this ongoing question about how involved should an instructor be in the annotations right so like. We have a number of folks who think that that's a student space and there's a really there's something something really nice about staying out of it letting the students annotate and staying out of the conversation others like to signpost things. And one of the things and I've been using it in my reticence sports class and I stayed out of it the first time though the class like I didn't annotate anything I just gave them the guides. And but I feel like one of the advantages and that's something I'm going to do in the future is to actively participate in the annotations after they've started annotating. Right not before I don't want to I want to tell them what to read but to engage in conversation with various students in the annotations and actually slow down the process. Because what happens to me all the time and this because again mean I spent my whole life reading and trained to read and know how to read and do it well I moved through text really quickly right and I moved through discussions really quickly and this is something I do as a teacher. And some students get it some students don't we just move on and that's sort of like I've always been kind of willing to accept that but something I've been doing the last couple years is really trying to do less do more with less and to slow things down. And I think that there's an opportunity with something like social annotation to more slowly and more thoroughly engage a reading to position key concepts from the course position key frames or perspectives. And then if you engage with students through that you not only create better feedback loops between you and them. But then it moves it very quickly from a completion activity or another task list inside of your LMS to was an experience at that point. I still think it's already a good experience for the students in the way that we have it set up at least as ensuring accountability getting to read the text getting to think about the text. I don't think it's problematic that way I just think that depending on how much affordance you have as an instructor or how much agency you might have over your course right there's there's those kind of considerations I am tenured faculty so I occupy a little bit of a different perspective than a lot of folks but But I think there's something to this and I don't know if it's better to annotate with them or not I don't know if it's better to turn into a series of things they complete or not and I just don't know but I now I want to flip and try the other side because I did this version. And I'm like OK I like that but what if I spent time arguing with them in the text right or pointing out things like you know like we read something like case a lemon and layman's piece about being in Oxford Mississippi and culture and race and football and and you know they missed some of the the significant elements of his story has been an African American male in Oxford Mississippi in a football culture he came from he's a IU graduate. And so there's you know like and so I win class we talk about it but it seemed like there's an opportunity to extend that more meaningfully in the text to pin some things down before that happens. But I don't know how it fits in my class like this is this is my challenge like I don't know what to do with it yet so. Hey this is great I know Justin in terms of you know you bringing to the four aspects of participation you mentioned accountability we've touched recent assessment. Nate can we bring criticism question up from the Q&A and maybe have folks just kind of riff on that a little bit if that's relevant maybe as one of our last questions are getting towards the end of our arrow here. So folks can can see this I'm curious if if our panel has thoughts on on a question about combining social annotation with I guess labor based grading if folks have a sense of what that means or how that is being approached or maybe other forms of assessment as well. Real quick what do we what do we specifically refer to in labor based grading. Is that contract based grading is that what we're assuming we can bring him up on stage if you want. I mean I want to make sure I understand yes you want to raise your hand and I'll bring on. Thanks for hanging hanging in there with us folks as we start juggling some tech here but this is this is kind of what it's all about Curtis welcome. Hello can you hear me. Yep. Yes, yes. I'll even now you can see me. How about that. I've loved the session so much. I'm also an English faculty and so the conversation and I am also as I said in chat engaged in a research project that I'm currently designing for the for next year. But I've been really interested in the work as out in ways been doing in labor based contract particularly to kind of really address issues of equity and and and relationship to white language supremacy. And so so it's a form of contract grading that as we've been talking about social annotation is sort of evidence of participation. One thing a sow does his he really has people keep a labor log so it's not about how many posts in the annotation but it's how much time are you spending reading how much time are you spending responding how much time are you spending. Annotating or responding to annotations and documenting it and learning the kind of thinker and reader and responder you are and documenting sort of in a metacognitive way how much time you're committing to the task. And and how and what it's doing for your learning and then combining that labor grace with also writing of essays so that we just extract grades from the entire operation. Yeah so that's why I thought you were going but I just wanted to make sure it's a diverse audience. So I saw his work is as a wonderful and in a lot of conversations and I do something similar and it comes sort of below but before his work came out which is this semester actually in my sports class I used the learning record method which is Peg Cyberson and Mary Bars work which is very similar to the labor base approach that I saw using which is students they document their progress to the course they have regular reflections rather engagements. Folks you know what they believe they're learning and they use those evidential things and the things they create to to make the case at the end of the semester like here's what I've learned and here's the great I think I deserve based on what I've learned not based on a performance or based on a particular outcome. So it's a little more like they have to argue for the grade rather than just keep track but but it's the same idea and that's exactly what happened in my sports class this semester was they were and several of them pointed out like you know on our annotation assignments. You know here's the kind of work that I did in those annotation assignments or they were taking the like one student said I only did two per post but they were taking me so long because I was really trying to think through my response which even if it's a lie I consider it. I bought it I'm like I'm in okay you're going to you're going to tell me that I'm in but so I think there's a lot of room for that specifically you know if you get into practice and students get comfortable with it. My problem is that it's the first time students have ever experienced anything like this. And the first time I'd actually taught this version here at IU I did a version when I was at UT Austin where Jeremy Dean was a grad student at the time and we both sort of you know work with Peg and a number of ways but it is a I found it to be a very useful vehicle to avoid this like I'm just doing annotations because it counts for a grade and it became a another piece of evidence in the demonstration they can make for their engagement in my course. And once they once we got over that hump after about week four or five when they started to realize what I was doing I wasn't grading anything I was just responding to their work right. Then I think we got a lot of buy in a couple folks dropped out but more folks really really bought into the process I think it's I think it's a rich opportunity for any of that framework. And but I'll let you know I'm going to do it again next spring and that class comes around and I'm going to do a little more personal participation in there but I think the framework is really well suited for social imitation as a learning activity. Well especially if you're going to have them know I'm talking about labor practice but then like their identity in relation to labor their positionality in relation to what they're doing what they're being asked to do how the languages is imposing on them or not you know those wonderful sort of distinctions that oftentimes get swept under the rug in higher education despite our commitment to diversity and things like that and underrepresented populations we still have structures that don't always make those things as visible and I think having students do that which is you know really core to what I saw as work is doing and helping us think about so yeah I think it's great I think it's perfect for that. Yeah, this is great. Yeah Chris thanks so much for jumping in here a little bit of improv towards the end of our hour but I really love it. We are coming up right on the hour. You know, I'm always game to kind of hang out and chat a bit although I also want to be respectful of everybody's time and I also just want to formally thank this entire team from IU for joining us today so although some folks may need to run. And maybe some folks can also stay for a few more minutes. I just want to thank on behalf of the conference organizers and hypothesis and the broader I annotate community. You know everyone for joining us today Mary Helen and Sarah and Laura and of course Justin really it's been a pleasure personally to learn alongside you for the last year. I look forward of course to our future work together but thank you for sharing again your wisdom with with everybody who's able to join us today. This session has been recorded. We'll be able to you know reference it you know down the road and again folks are welcome to hang out for a few more minutes if they're available but again you know we're at the hour and so this is you know the end of actually the Educator office hours part of I anti but it's also really kind of the end of I annotate so I guess we've. We kind of made it all the way now you can put us together now how about that. There actually is there is one more session if you still have appetite for more immediately following this where it will be reflecting on the week as a whole. I know people are probably too tired to do that but unfortunately I have to go be the taxi for my kids okay Uber dad but not in the way that that sounds good right. But I really appreciate you all having us on and let us share our story and talk with folks and if people have questions about what we're doing or want to know what we've done and. I mean the data will eventually be parsed our research will publish but if you just have questions about a programmatic design engagement or the way we teach with things. I'm sure I speak for all of us that feel free to reach out email me and our email any of us or I can connect you and we can have continue our conversations after this.