 Welcome to liquid margins. This is social annotation strategies for large courses. I'm your host, Franny French. Today's guests are Melanie Walsh. She's assistant teaching professor in the information school at the University of Washington. And go Huskies. Scott Johnson, professional assistant professor of history and undergraduate history program coordinator at Texas A&M University, Corpus Christi. And Kevin Richards, assistant professor of teaching clinical outreach coordinator for the Department of Germanic Languages and Literatures at the Ohio State University. And then our moderator today is Jeremy Dean. He's the VP of education here at Hypothesis. And with that, I'm going to stop sharing and turn it over to Jeremy. Thanks for any thanks everybody for being here. Super excited about this conversation. Best part of my job is talking to instructors and students about social annotation. So this is always a great opportunity to do that. I'm especially excited though because we have introduced some features and some practices that better enable large courses to use hypothesis. I think everybody here is using hypothesis. And Canvas actually, maybe except for you Scott, are you on Blackboard? All right, so Canvas and Blackboard, but we're here really to talk about hypothesis and how it can be used in large courses. That's that's the focus, but I want to start generally and just hear a little bit from each of you guys about what you teach and a little bit of your teaching philosophy. Then we'll dive into the social annotation of it all. So maybe we can start with you Melanie. Sure. Thanks for inviting me to be here. So I teach data science and data ethics and digital humanities classes at the University of Washington. This quarter I'm teaching our introduction to data science class for the undergraduate informatics program at the UW. And so we do a lot of technical work with data science, but we also do a lot of readings, which is where I incorporate hypothesis. I'm teaching a smaller masters of library science class where I'm also using hypothesis. I now use hypothesis in all my classes because I feel like it's the best way of facilitating discussions about readings that I have ever found. But I've used it in my other classes that I teach as well, including information ethics and policy. And I guess a little bit about my teaching philosophy, at least when it comes to data science. The main book that we read in the data science class is called data feminism and so there's this commitment to engaging with ethical questions and questions about justice and not only to have it be strictly about data science so doing those readings and having that kind of like, you know, engage discussion with the students and between the students is a big part of my philosophy I would say. Let's hear from you next Scott. Hi. Thanks also for having me on here. I teach a variety of history courses. I'm trained as a French intellectual historian. And so, at least from where I come from in my teaching I have a big commitment to analyzing dissecting understanding texts. And that was one of the big initial draws for hypothesis for me, especially at the beginning of the pandemic to get students annotating working through texts from home socially distanced. So the main sorts of courses I teach here at A&M Corpus Christi is a big learning community history course. So history course paired with writing studies and then seminar. And in my history sections I have up to 175 students, depending on the semester. So one strategy I found really helpful for that really large type of class setting is to have all the students do their reading through hypothesis beforehand and so they, before they get in the room they've already had discussions with each other, marked up the text. And then when we get into class we can then break things up in the different interactive activities sometimes using hypothesis in the classroom and then sometimes doing other things. Kevin. Hi, I'm Kevin. I teach in the German department. So some, and I've kind of become like a specialist for their GE courses and kind of increasing enrollment, moving those onto distance learning sort of styles. And so one of the main questions there for me was how do you get someone who is taking the course not to feel like they're kind of alone at their kitchen table. How do you like connect them with others and so been using quite a few different strategies and technologies to do that and when hypothesis then became sort of at least it was sort of like an experiment I think that last semester is offered. And so I wanted to try that out and it looks it looks like it's done a really good job so I use it in my fairy tales course we have about 300 people in that. And I break those into groups with canvas, it makes it pretty easy. Yeah, so they're able to not only like kind of have discussion about the texts, but also they're able to kind of develop their, their thesis, or like take like use notes from these annotations to develop sort of their papers that kind of come then after after sort of like five weeks each five session, or five weeks of section. But yeah, I think that's that that's sort of like just in general, what what I do. That's great Kevin, I'm going to keep it with you and ask each of you guys to tell us a little bit more. So I the reason you were chosen to speak today was partially by me coming through the data of our current cohort of folks using hypothesis and really looking at course size. And what were the largest courses using hypothesis and and being the most productive in that regard so large courses with lots of annotations essentially. So what I'd like to do is here gesture towards this Kevin tells a little bit about and you know, Melanie complicated by saying you're teaching different courses with hypothesis so that, you know, seminar style course didn't hit my radar, but you know, it's there. Tell me about the course that you're using hypothesis with the topic. The size I think we got that from you Kevin, but then also a little bit about how you're, you know, integrating hypothesis specifically what you're asking students to do so maybe if you pick it up with that last part and then Melanie and Scott we can hear a little background on the course as well. Yeah, I wanted, I wanted students to sort of like, you know, have it replace not entirely replace discussion but add a second sort of aspect a second sort of area where they could have that kind of engagement with each other. And reading, especially if it's online can be like another place where they feel isolated. So having social annotation is really a great way so they can see like an engage and get different perspectives connect with each other and even what each with each other and even when it was in person. So I had the sense in like large lecture halls that the students were only getting the perspectives of those people who were like next to them if I like asked them to do something in class, and then readings like completely not so I think even like if this were in person. I'll continue with the social annotations for the readings. And so what I've done is I made sure because it's a fairy tales course I thought that that would be sort of the easiest way place to implement it because most of the texts that we use are out of out of copyright so I could pretty pretty much take translations update those translations makes way presentations out of those so it also is a little bit more presentable and broken up for their reading, and then, going into canvas. And then create, use the tool to create, create sort of the assignment and then divide, create discussion groups or reading groups, probably about eight to nine people per group. And, and then that would just divide those up the very, the very first time I went through and kind of provided sort of a model for like what the expectations were for the annotation and provided sort of like an example. And then did that for all the 37 I think 3738 groups that there were there. So that's a lot that's a lot to kind of manage like all those groups. So I did that the first time. It would be nice if there's a way to kind of like maybe there is and I haven't like found it to, to provide one example that goes to all the groups that I have in the in those, you know, so that would just save time and maybe I would do that then for each reading and kind of provide sort of like an example or question or direction. And so I then considering like the massive amount that would be required to grade something like that mass amount of time, I would, I set them into like five week blocks and that's sort of like just how I break up the semester. And then in a distance learning course, it can get kind of tedious, especially in the spring, where people start getting like, sort of exhausted around week 10 to break those up where we sort of like renewed sort of section that we're doing each five weeks. So four weeks they have readings, all the readings are socially annotated and then the fifth week they can catch up on any of those readings. So if they missed some of them they can have like a week where they can go back and like fill in the gaps and that's also the week where they do the grading. And to grade that I will take from each week just a sample like a random text. Usually it's the text that I think is the most difficult, you know that maybe like that they need or the most important one. So like how they've, how they've annotated on that the requirements for me are that they have at least three sentences, questions, they can be questions or responses but they should be three sentences. And they should be a substantial content, not just like, you know, like, hey, this was great. What color was the hair of that character, you know, things that like really get, get more to it. It reduces the amount of like time I spend in, in actually reviewing what they're posting and so I get sort of a sense for their grade and then I put that in for that. And I do that so every five weeks. So I have like sort of like a little break in, in the other other areas of where my attention would be. But I think that's sort of like is a very general sort of outline of how I how I use it. And it's been encouraging to see like their engagement. And people are kind of off, I will like just say hey if you go back and redo this, they get the practice of it and then they know like what they what's expected going forward. That's great Kevin. I love the point about reading can be isolating. I think anybody, I certainly felt that at times and fortunately stuck with it but I worry sometimes that that's the time when we lose some students when a particularly dense or difficult reading is in front of them and they have this community to support them and some kind of interaction to help them through that they may drop off at that point. So again just to review what the courses you're teaching your largest course which are one of the largest when using hypothesis and the topic, the size and a little bit about how you're using the tool and I think you got all that Kevin, I'm pretty sure. So appreciate it. Thank you Melanie and excellent Scott. Sure, so the big class that I'm using hypothesis in this quarter. It's called foundational skills for data science there are 200 students who are in it. And in this class. I have each week, one reading that the students are required to make two annotations on. Similarly, they're required to make a substantive annotation and this can be a comment or a question or a reply to another student's comment which is I think the students really enjoy so I'm really glad to be able to include replies to actually get that kind of dialogue going that's one of the big motivations for using hypothesis for me. In the past when I had done canvas discussion boards or you know like use WordPress to have students try to have a discussion in a blog forum. Students would just like repeat the same points over and over again after a really complex dense or interesting reading and it was really hard to like encourage them to engage with specific passages and that's able to happen much more organically I feel like in hypothesis where like you can really just zoom in on particular passages that you're interested in. So, so yeah to substantive annotations and I give them like a whole kind of buffet of suggestions about what that might mean like you can make a connection to one of our lectures you can make a connection to another reading you can bring in your personal experience as long as it's done and such and such way so I try to provide some guidelines for for what's required. I suppose the way that I use it oh so then for the large lecture courses what has been essential for me is similarly. We have eight different sections that the students are in they have lab sessions every week so I organize them into those eight sections that have about 25 students. So the students are only annotating within their section and they can only see this the annotations from those other students. I'm kind of curious about what it would even be like to have 200 students annotating one reading. I think that I've just assumed that it would be far too chaotic but I think it could be kind of an interesting exercise in some regards but. Well, I don't mean if someone else is doing that I'm very curious to hear if you're doing that. But yeah, just 25 students, a piece for me and the main way I've been using it is just to. I basically review it as I'm preparing for the lecture and I can see like what are the big questions that students have what are the big areas of confusion what are students excited about. I think it would really help me like have them lead the discussion or like know what's going to be kind of a vibrant discussion in smaller classes I'm also able to like note down. Okay a student made a comment about this or I'll remember that a student made a comment about this and I can like, you know, reference them by name, if the discussion is kind of flagging and be like, you know I remember that you, you know had something about this can you elaborate on this point that you made in a larger lecture course I think that's a little bit more challenging but I can still kind of sometimes remember specific things that I know students were talking about and I can say, you know a lot of you were saying, and I like doing just being able to say like I saw a lot of questions about like this, and it is like one way of like acknowledging like I'm, I'm invested in your conversation to. That does lead to one ongoing question that I have about the way I'm using hypothesis which is that, like, I have wondered to what extent I should try to be replying to their like actually be textually part of the conversation in the margins like in a large lecture class again. That's sort of impossible for me to do, but I've thought about should I have my TAs for each section, kind of jumping in more and say and like really being active so anyway that's an ongoing question. And then in terms of grading, I have the TAs grade the annotations each week, based on that kind of substantive like annotation rubric. And then there's Canvas and SpeedGrader, it's pretty straightforward. And I think it goes pretty, pretty quickly for them. So yeah I think those are the, that's like the main way I'm using it in the data science course. That's great. I actually had a couple of follow up question a follow point before you get to your, tell us a little more about how you use it Scott. So should I be part of the conversation. Basically, that was one of your ongoing questions. Did you mean that sort of pedagogically like should I let this conversation sit and see how they're interacting and not interfere. Or is it good for me to be present like just my pet it was that more of a pedagogical question like how much of the teacher be part of this discussion. Yeah, I think it's a pedagogical question and that's big because sometime, like, I do think it's great that they, because at this point, I don't often. Actually reply to what they're saying in the margins I mostly just do it in the classroom or in discussion. And I do think it's nice to like let them have their own discussion with each other, so that I'm not like, you know, sort of disrupting that like camaraderie that they're building together but sometimes I want to like affirm or acknowledge that someone's made a really good point or sometimes I see some like a student saying something slightly problematic that I kind of want to like reply but then I don't I don't want that to be the one time that I'm jumping into the conversation and make it feel like this student is really being called out like in particular. So, yeah, I guess it's a kind of array of things. Right. And now I just want to circle back to one thing Kevin said I mean it takes all kinds it takes all kinds kind of situation right different texts different pedagogical moments where the teacher should be present or should be interactive and one where it's nice to step away I think it does all situations Kevin was mentioning maybe pre populating some texts with comments. You know signposts or something like that or a prompt that that would help guide students or elicit a response from students and so we see that as well we could get into those different approaches and I did by the way Kevin just want to register. I just want to make sure request embedded in there which is, if I've got multiple groups, and I want to post such a signpost or question, I have to do it, and times right now with hypothesis and be nice to ship my annotations to all those groups so that was registered just want to make sure you got that. But there was one other thing you said, Melanie that I wanted to follow up on. And that was, and forgive me for sort of testing a market testing a kind of slogan for you but one of the things I've been thinking about with large courses and hypothesis and both. So far as I'm hearing it you guys are using in this way is does the hypothesis canvas groups or blackboard groups integration. So, you know, those larger courses in some ways to feel more like a small seminar in the sense that you can sort of, as you described perfectly Melanie like with a smaller group, you can call people by name and really you know bring them out and honor their their points of view and, and elicit more, you know, depth from their from their ideas and obviously the more people that are in the room, much harder it gets to do that. And I do feel in some way that this tool is making that larger course a little bit more intimate in that sense, I don't mean to totally prompt you too much to say yes to my marketing spoken but say one more thing about that and then we'll turn to Scott. Yeah, definitely I think like intimate is a word that was like in my mind before this conversation. And again, like, especially in a data science course when we have a lot of technical material to cover but we also really address these more conceptual theoretical questions ethical questions and like give that, you know, it's do I think that this is one way that we've been able to achieve that to some extent. I do think like another way. I don't know I I've just been thinking about how I should probably be incorporating my TAs more into my thinking about this and like potentially have them engaging with the smaller smaller sections in the lab sessions. Again, we have a lot of technical material to cover so that's kind of specific to like my class and how it's a little bit challenging, but yeah I would say intimate for sure. Yeah, and I guess another aspect of that is knowing where the group is at or knowing the different places the groups that it's much easier with a smaller class you're like okay we're all can use what this concept was zero in when you have you know through your students it's hard to know where are people struggling and maybe that comes a little bit more visible and is with this tool. Alright Scott, we're wanting a little more about your course, the size and as Melanie and Kevin have done a little bit about how you're using hypothesis in that course. The main course, the semester the big one is has about 120 students and it's modern US history. So from reconstruction to the present. And it's framed as a history of the present course. So explicitly ditching a textbook narrative and focusing on big themes like structural racism, Royal the United States in the world identity inclusion belonging immigration. And also changes in economic thinking in the United States. So, because of that framework. Each week they have very specific readings from different experts that I'm able to pull through universities institutional databases ebooks journal articles that sort of thing. And with hypothesis, I get them to engage with those texts ahead of time, before the class begins in all the ways that Kevin and Melanie have suggested substantive annotations, although I've become a little bit more lenient on what counts as substantive, because I want to get students to think of the text is sort of hypertext. So I have some students that absolutely love picking out old people, historical figures, and then finding their pictures are finding videos of them. And I'm beginning to count those more and more substantive in different ways. And a couple times I have had to email those students and say, you've done this three weeks in a row, maybe try to switch it up a little bit. But I absolutely enjoy that. And especially when I get students to think critically enough where they can generate their own meme on the content. I think that's fascinating. Because it gets me to engage with what the current memes are, enforces me to think differently and get inside my students heads, but also because it shows them processing and trying to interpret the information and creative ways. I, unlike Melanie and Kevin, actually, still this semester have stuck with the large group format. And so all 120 students are in the same room, discussing with each other. And when you at the end of the annotations that week, when you look at the text it can look like a mess. But I explicitly walk students through how to hide annotations use the little eyeball feature, if that's like too intimidating or too confusing, how to keep annotations private for themselves in case they want them for their notes. But then because there are so many different seminar sections that have their own community in the learning community. I want to use history as a way of bridging those sections together and so to think of them cross section cross study group as a as a big point of discussion. And I think in coming semesters I will probably play with the small group feature just to see how it works to compare it. But as of right now up to this point I have sort of been happy in the chaos, and I think the students are happy to see what people in other sections are thinking and talking about with the text. Happy in the chaos, I love that. But it is I'm really glad you brought this up Scott because you know I did notice this in the data right and that's part of why I was really glad that you responded is that Melanie and Kevin are leveraging our small group functionality and canvas which also exists in blackboard so they can break up these larger courses both of them into you know eight or nine groups for those 300 200 person courses and then they really do end up roughly the size of like a seminar but you're, you're doing what Melanie was wondering about like these texts. You've got 120 students, you're happy in the chaos and I think you point out one of the really great reasons Melanie if you're wondering why sometimes is that, you know, connectivity across section, right when you have all those. I mean that's, it can get messy and I think it's so interesting that you teach people to navigate the mess and the chaos, Scott, but there's opportunities for connections. There's more generative more generation of ideas when the more people are there as long as you can, can navigate it. I think that's a super interesting contrast here but I'll be interested to see what it's like when you try the blackboard groups functionality which we also have and of course you can go back and forth between having well this text we're going to do it all together. We're going to come in this text we're going to do it in groups and in fact you can do some texts both ways. Right, you can sort of like let's all dive into this as a course, and then maybe break up into sections to read it differently or read it again or show what we got from the chaos. Scott, in the chaos though, are you happy grading the chaos or are you not grading the chaos. In part because I like using hypothesis and that diagnostic way that Melanie discussed. So seeing what questions students have before we get into the classroom that week to sort of fine tune what we're going to discuss and work on together. And so I really want to go through all of those annotations myself. It can be form of braiding, it can be rewarding as well as exhausting. And I think I'm just well trained and getting through the exhaustion. But I, I do think that there are ways that I can improve getting through both the grading as well as sorting through everything and those diagnostic ways. So after I tried to get my students to use a hashtag. So every time they wanted me to explicitly respond. They had a pressing question. I wanted them to either hashtag professor or professor reply. It didn't really work yet. So I think I need to be more mindful about getting them to think about that and play around with it. Yeah, I, I partly want to be there grading so that I can see everything explicitly. It's, it's labor on their part and you want to honor that. Really interesting point about the hashtag. And I think Melanie if you if you do venture out into the full group. You know, using a tag can have a number of different benefits right you're talking about adding questions so that you could search for questions and make sure that any questions that come up are answered by you but of course on any reading there might be sort of different. Dematic or theoretical approaches that you might want students trying to foreground and they could use, you know, this group is in the 120 students or 200 students is going to do the feminist approach and this. This group is going to do the Marxist approach or whatever and they could track that by those by tags, even in a large group. I want to open it up for questions before our hard stop at quarter of. So I just want to give each of you a chance to say one, you know, last sort of share of your thinking about hypothesis and your teaching. There's something that hasn't come up in this conversation that you'd like to share or, you know, request of hypothesis since you have our ear and we can be held to it publicly that you've made a feature request, like Kevin did. You know, last thoughts before we open it up to the, to the, to the large audience that we have here and some questions that are already in the chat. And Kevin, let's start with you just any final thoughts. Final thoughts kind of like some things that jog my memory to I thought, really, I want to just reiterate also something that Melanie had mentioned is that you get like sort of you can take sort of like the temperature of the course. And through like looking through the annotations and so we do a live session, mostly it's asynchronous but then there's a live session every week at the end of the week. And so using the annotations you can really see like what's what what could lead the discussion what could be some really context to talk about what's being maybe misunderstood or what can generate more more dialogue and discussion. So yeah I just wanted to like yeah, I think that's also a very good way of using the annotations and what they can offer. I did just want to clarify one thing. And is it correct to say Kevin you're fully online with what you're teaching right. So the two courses I teach a teacher teach a Holocaust and lit and film course that's also DL. And then this is second and so trying to combine sort of like the opportunity for it to have like those that live sort of synchronous discussion but also have most of it be asynchronous. That's, that's the way that I'm teaching at the moment. And Scott and Melanie your courses do have a face to face aspect, or you also online. We're in person. This semester in person. I did, I get to get ill at one point so we had to switch online and hypothesis was great for that. So Scott since you have the mic final, not finalist thoughts before we open it up to the, to the audience for questions. Any reactions to what your colleagues here shared or any jogging says as Kevin mentioned. I think it's great one that three different different types of courses at different institutions with different students and formats are all using it in very similar ways I think it speaks to the flexibility and adaptability of the tool. At this point, and I don't want to sound. I don't have having the teams egos too much, but all of these sort of things that I really wish hypothesis could do when it first emerged a couple years ago. I think the team has actually been working on with the one, one big request being tagging video. And I know that's sort of the, the next big step but also as someone who knows nothing about coding, I can only imagine how difficult it is to actually kept working. Thanks, Melanie. So sort of building on that point actually I feel like my final comments are kind of in the vein of like, sort of in the vein of a feature request but one thing that I thought. So in the data science class we're mostly just reading chapters from an online, a book that has been published online it's like PDFs or it's the chapter online but in some of my other classes. I'm able to have students annotate YouTube videos by having the transcripts next to the YouTube video through the doc drop.org link. And we've also had some podcast episodes that have transcripts and that's been really fun to do so I've started to get excited about maybe in the data science course as well incorporating other kinds of media for students to be able to annotate and I think another thing I was thinking about is annotating images which I think has been another feature request and I've thought about like, you know it'd be great to have a data visualization that the entire class could annotate specific parts of it and ask questions so I guess. Yeah, one thing for me is like continuing to think about how I can get creative with what how we can use this tool. I mean, there's some great ideas here, even though you are using it in some other ways and I did just want to shout out Scott about that idea of using images and memes and things in video in annotations. I'd love to do for any session sometime I don't know how we choose the people for this but maybe Scott could come in and speak to his, each one of you said something about substantive. And I bet if we gathered your definition of that they'd all be a little different. Scott was sort of suggesting that he has a, maybe a more expansive one like it could be an image. And I totally agree with you and I also agree that like, if you keep just putting the image of the author like I want a little bit more at some point but it's not a bad start. So maybe an episode on what is a substantive annotation and certainly would like to call your definitions here but for any any questions that have come up in the chat that we could foreground here. Yeah, and some of them have been answered thank you to the panelists who already answered a couple of the questions but Ying Fan wants to know what prompts are you using for the data science course for annotation. I saw that question the chat and I really appreciate it because I'm not. To be honest I'm not really using prompts at this point I liked the, the suggestion about being able to have like a prompt populated in each reading. So, yeah, in previous courses sometimes I have had quite specific questions that I wanted students to answer but the way that I've been using it so far has just been like letting students kind of lead the discussion and find what they're interested in and I basically just discovered that I didn't need to be providing prompts because they are able to like really latch on to what what they are interested in but I do think that it could be, it could be useful to bring back in to like have a more guided discussion potentially but no no real prompts at this point. I think a lot of it is in that how you define substantive, you know, like I want you to annotate so what does that look like it's not necessarily a specific scavenger hunt it's kind of a methodology, right I want you thinking, writing, reacting to the reading in these types of ways, rather than search for a specific thing or answer a specific question. Yeah that's well put. And then Manny Fernandez. It's not so much a question I don't think it's just a comment but Manny can let me know if I'm reading that wrong but using a tag group AB and see and seeing which group tag the document the most participated the most would be kind of fun for freshman class within the guidelines of course. So, I don't know if you want to roll with that or not. You're all, you all are here because of the voluminous annotation of your courses so quantity did matter for your attendance here but I'm not, you know, do you guys put a number on the annotations when you somebody said they say three annotations right you guys put a number on how many and do you ever have groups compete. I'm competitive. I think what's kind of interesting is when it's I have like a minimum of, there's quite a few readings their fairy tales so there's we cover quite a few of them in a week and then there's one scholarly article they also have. One thing is like there's a minimum of one per reading, and it should be like three sentences, but some people go way beyond that. Like they, they really use it as a note taking function, and then also respond. So, you know, in, I mean it's about the third of the class that goes beyond. And yeah, it's fine that they are just like very much more engaged, in part because of the annotating feature. Yeah I require students to make two annotations but I love when they go beyond that and I don't think it's competitive I wouldn't describe it that way. For me it seems more like, like, organic and communal that students just want to say, oftentimes it'll be smaller things since they're not required but like, lol ha ha this makes me think of something and it's it's really like feels more like an actual conversation so yeah too required but I love when they go beyond. I love that Mel and it reminds me something you said earlier right that I don't want to take put words in your mouth that sort of basically that this is a little more organic than the discussion form in terms of generating conversation I think that's true because of the textual reference that everybody sort of grounded in but also just in the way that people are responding to each other. And you said something about there's something about discussion forms that sort of suggests everybody repeats the same thing. And maybe just the way that conversation, you know, literally is sort of tracking from up from particular piece of text and then and thread of conversations just, you can't just repeat. There's it's going in a direction and you have to follow that direction and continue it or taking in a new direction but you can't just say the same thing. Which by the way, would happen if somebody in the chat said like, can you have everybody hide their annotations first and then reveal them later. It'd be an interesting thing and of course you know then you know Kevin wouldn't be able to just you know, repeat or crib off of what Scott said but of course what you'd end up having is 15 people all being like, this is what's happening here instead of one person saying this is what's happening here and this person saying they disagree and the other person saying I agree with person one and it turning into an actual discussion of the meaning. Scott I want to give you a chance to respond I'm not sure where this question went in terms of volume and rubrics or do you have a direction to take it in. I say students need two to three annotations, and then I give them bullet point suggestions, asking a specific sort of question, showing their analysis responding to someone else's question that sort of thing. But I think if I were, if I were to ask students with everybody annotating in that big class to do more than two or three that I think it might be become unmanageable for me. And even more KSE. Yeah, and like the other people on this call I, there are always students who end up with like 20 or 30 annotations, which is great. But if I, I think if I made it a competition in that large format. I think I would probably drown in the deluge of comments. There's a good question just to end up here George asked, you know, how do you deal with the repetition, right of people saying the same thing. Are you on the lookout for that or is it organically being avoided. I think, or do you have to intervene. Good. I think what's kind of interesting is like the often will highlight parts of the text, and then comment next to it so they're not really unless we're responding to that highlight. They're not really repeating sort of the same sort of comment because it's really it's associated with that part of the text. I haven't like encountered that. And that's kind of amazing that I haven't encountered that and it's such a large group. And I think it's because it's tied to specific, you know, sentences in the text with the highlight. You can see this real estate is taken and I need to kind of build on that real estate or else claims a new real estate. I know somebody had a hard stop so I'm just going to cut us off there and let Frank caucus out I could keep going this is a really enjoyable talking to you all. I'm 45 probably going toward 946 any second now so I just want to thank everyone here for joining us today. This was such a good discussion always just goes by so quickly. So thank you Melanie Scott and Kevin, and thank you to Jeremy Dean such a good moderator and Aaron holding down the chat and Nate as well and Becky was in the chat to colleague so there will be a recording of this hopefully that'll be available next week and everyone who registered for this will get an email with the recording link. So thank you very much for coming to liquid margins today.