 Welcome to liquid margins and this is a tale of two semesters evaluating social annotation at you men, which apparently they don't call it you men but we needed to shorten our title so apologies to University of Minnesota right off the top. Okay, and today's guests are Shawna crossing. She's a spatial technology consultant in you spatial at the University of Minnesota and the dash program associate director. And she can probably tell us what dash stands for. And then Karen Jeanette she's an instructional designer at the University of Minnesota Duluth. And then our moderator today is Jeremy Dean VP of education here at hypothesis, and my name is Franny French, and I'm your host, and I will turn it over to Jeremy, and I will see you toward the back end of the show. Thank you again so much for being here, and I'll see you in the chat. Welcome folks. Very excited to be here. I've been working with Shawna and Karen for many years now. They are the folks that brought hypothesis to their respective campuses to Duluth and to the twin cities and it's been a great collaboration. So I'm excited to have a conversation today but today we're actually going to do liquid margins a little bit differently Karen and Shawna have been working with data from the hypothesis usage through the pilot and into subscription phases and have surveyed instructors and students and looked at the results and looked at other data and are going to share that stuff with us to start off. So I'm actually going to hand it over to them to take that away and then we'll follow that with a conversation. Thanks for being here. Thanks so much Jeremy. I'm Shawna crossing and I'm going to we're going to get started is that sharing working. Good. Okay. Hello everybody thank you so much for being here with us today we're carrying and I are really excited to share this project we've been working on like Jeremy said, I've been working with Jeremy for, I don't know how long, different things couple years several years. And luckily been able to work with Karen as well, and we'll explain a little bit about about about what we've the project we've been doing and really would like conversations so Karen and I'll try and rush through this quick and would love to have conversation and reaction and responses. Again, my name is Shawna crossing. When I started this project I was working as an academic knowledge in the College of Liberal Arts at the University of Minnesota Twin Cities campus. I have switched positions now I'm a spatial technology consultant at the whole different ballgame but stayed on to finish up this evaluation project. The dash thing if you care digital art sciences and humanities so all sorts of great fun things. Yes. Hello, I came into this a semester after Shawna started it. I have been the lead contact at the Duluth campus to pilot hypothesis. So working with Shawna to develop an evaluation has been a treat to say the least we've learned a lot of things during this time. I don't think I need to say any more right now but I'm just excited to share our findings so I'll let you go Shawna. Alright. So again, one thing a caveat I want to say Karen and I are academic technologists, we are not researchers. So this project was done as an evaluation of the tool, and his evaluation of the pedagogy to help inform us this was a new tool to us to help us inform how we were best going to support instructors and faculty in using this tool this is not. I'm going to be very clear this was not a formal research was not formal research we did not use research protocols so just keep that in mind. All right, let's jump to the good stuff. Hopefully. Okay, a little bit about the University of Minnesota. We have five system campuses. The Twin Cities campus, the College of Liberal Arts which is where this pilot took place has about 14 students, and you can see up here. UMD Duluth campus has about 10,000 students and they did it system wide so it's just a little bit about what we did Rochester campus also piloted but they didn't really participate in the evaluation. We included this slide just as a refresher of what hypothesis looks like in a Canvas course with the LTI the integration. For us, the reason we piloted the, the integration was to make things like sign on and logins easier for students, easier for faculty to set up and the grading was a big deal. So this was just part of this is more just a reminder. The pilot, we started the College of Liberal Arts started the pilot in the summer fall of 2020. It was part of the emergency remote teaching that we all lived through the Twin Cities campus was the only one doing it we had about 25 active courses and I have to give a ton of credit to John Chen, who I know is presented here in his grad students Jinran Zhu and Hong Shu for their help in getting this going spring of 21 Duluth campus joined us in this still continuing remote teaching response. You can see the number of courses about nine at UMD about 30 at the Twin Cities campus and fall of 21 was just more a regular usage. And you can see the course is about 10 at UMD and about 50 at the Twin Cities campus. Our evaluation again, this is evaluation. This was not formal research after fall of 2020 our first semester I did an informal student survey and interviewed faculty along with having conversations with faculty along the way so really more anecdotal. In the spring of 21 we did a student survey we had about 65 responses, and we did three faculty focus groups with nine faculty total which were really, really impactful I think Karen and I both almost very impactful. And fall of 21 we did a student survey with 89 responses and a faculty survey with, I don't know why that says 18 because it was really 17. And that's kind of what this presentation is based on. Quick acknowledgements. Dr Bodang Chen and the grad students Jinran and Hong I have to give a ton of credit to. And I also want to acknowledge Remy Khalir and Ontario Garcia and Remy I saw was here. You're wonderful amazing book. This was not out when we started this evaluation so and we'll talk about this a little later. So, we would do things differently had this book been out but it doesn't really reflect this book at this point and that's one of our future goals. Okay quick basics and then I'm going to turn it over the good stuff we found in our survey that about 65% of the students had never used a social annotation tool before the semester that they did the survey. It was interesting about 35% had used it. And of that 22.8 was perusal 3.5% was some in unnamed tool and 28% was hypothesis. We do not know if students answered, like the same students answered semester to semester students might have filled it up for two classes we did not include those kind of controls. So, that's what we have. Oh, I forgot to fix that slide. So, how difficult or easy wasn't to learn hypothesis. We wanted to know how students were interacting with the tool. Fortunately, we found about 86% of the students found that it was relatively easy to learn how to use hypothesis so so that was, we were relieved to hear that we'll talk a little bit more about that later on Karen. Can I see I'll try screen sharing one more time. There we go. So, we wanted to be able to see how instructors were using the tool in class reported by students. So, one of the first questions we asked was what kind of uses were students seeing. And so our top five list is that number five students were using annotations to help with assignments such as reactions papers and other writing assignments. Number four, they were referring to annotations within class discussions and those might have been class discussions in person or in zoom. Number three, they were asking each other questions in hypothesis, or discuss the readings in hypothesis. And as you might expect number one students posted their own questions and reactions right in the reading. We wanted to know what kind of functionalities students were using in hypothesis, and we had about six students report that they were using or linking to images, video and multimedia. So that might be as expected. Again, some of this was during remote remote emergency remote instruction and I don't know if there was a lot of capacity to try a lot of new things. But we think that there's room for growth to linking to images and video in the future. Additionally, most were more students reported using groups and tags than images and media, especially once hypothesis made the canvas integration. The group featuring canvas integration work with hypothesis. We saw more growth in that tool. But generally speaking in our survey, about 22 students reported breaking into groups and about 25 reported using tags to identify topics. So one of the uses for tags. We did have some classes use tags for identifying the instructor. If they had a question or needed help. So students could tag their annotation with instructor, or they might tag their annotation with a role such as if they were supposed to summarize something in the reading for a group. They would click summarizer, and then all those annotations would summarize. So those were some interesting uses. We think that there's room for improvement there and that perhaps in the future we can work with instructors to think of other uses to use tags and categorize annotations. Next student survey results the interaction part so as academic technologists and instructor instructional designers we often think about four types of interaction when we're trying to create positive learning experiences and we keep in mind the student to learn or face interaction, because the students are going to go through something to interact with the material in online environments. Then we think about student to content student to student and student to instructor and so we were mindful of these interactions. So keeping this in mind we actually focus primarily in the evaluation on student to learner interface student to student and student to instructor so you'll see that reflected in the coming slides. We asked this question. Hypothesis is designed to help students understand class readings by encouraging interaction around the material and how helpful were the following aspects of hypothesis. So we'll share those. So first we wanted to understand the tool how helpful was it to have hypothesis or find, excuse me. I'll just start right over students found the proximity of hypothesis to the reading and discussion, very helpful. 80% of students found it moderately or very helpful to have annotations and proximity to reading, but 95% found it at least slightly helpful. Quoted that it forced everyone to closely read the articles which caused better class discussions. So students found the ability also to ask questions before class helpful. 85% found it at least slightly helpful but even more found it moderately to very helpful to ask questions before class. Additionally, we asked the question, how was having guidance from my instructor, how helpful was it to have guidance from my instructor about the reading. And 99% of students said it was at least slightly helpful to have this assistance from instructors. So what does guidance look like well some of our instructors were annotating the text ahead of time. Some provided rubrics for the quality of conversations and annotations, and some were simply just in the annotations answering and responding to questions with the students. We found that instructors were pretty intentional about creating active learning strategies around reading and signal to the students that it was pretty important and so we think that gave students also quality interactions with their instructors. Moving on to interacting with classmates. 96% of the students found it at least slightly helpful but more so even moderately and very helpful to interact with classmates. Some of the students said, nice to have classmates feedback on my questions in the text. It's nice to see what everybody thinks about the reading while also seeing the teacher's comments. Those were a few quotes that we thought were fairly representative. And then again, it also really forced everyone to closely read the articles which cause better class discussions, and it made the reading more interesting and engaging. Another consideration we wanted to know about was how did students find it helpful to share their knowledge and experience. And 95% of students did find it helpful to share their own knowledge and experience in the focus groups, faculty said students shared their own experiences and the annotation that they wouldn't have otherwise. And students sharing were sharing prior knowledge and experience and felt it was somewhat safer because it was tied to the reading and not a bigger idea question or concept. So I'm going to switch back to Shana who's going to take another section of slides. All right, we're going to jump into another sort of look at comparing these two. So we had the advantage of having multiple different types of places to ask questions we ask questions both of faculty and a student so this is a space where we looked at the two of them together and Karen and I at least came up with an editorial comment we'll talk about that. So we asked students, we wanted to know how impactful was it to them for them to have hypothesis and how they understood the material for the course and the content. And so we asked this question 6562% of them said it was somewhat more and much more. I thought this was interesting, much lower number was in the much more. And this was higher in the somewhat more. And about 34% of students said it was about the same. So we asked this question. After, well, we asked this question in both spring and fall. But when we were looking at these results we were also talking to faculty, and we heard things like this from faculty, I can see what they don't understand. And the faculty said you start class knowing where they struggled with the text, you can. Otherwise, you know the faculty would know where they thought but this way they knew exactly where they were so they could prepare better and address better. So, and I will say that these are the way that this is what students did say they saw faculty doing even though they didn't think it helped them understand better. So this is what they saw faculty doing that faculty referred to annotations and class discussions that they so that that in discussions like in person in class whether it's zoom or in person that faculty referred to this discussions. They faculty added information and post questions in the reading to guide reading to learn reading. They responded to student annotations. This was all big and we'll talk about this in a minute. So all these things are what students were observing, yet they didn't think it really impacted their comprehension. So this is where I'm going to pause for Karen and I we have an editorial note this was just a huge jumped out at us the whole time when we were looking at this. Students did not necessarily think they were understanding the material better, but instructors were adamant and I would say in the faculty, the focus groups were almost giddy. It's the impact that hypothesis had on student comprehension and this was material that they taught many times before, and many of the instructors involved in this commented on how much better the students comprehended the material so we have sort of a dichotomy, you know, kind of a contradiction between what students thought and what faculty thought. We did know that faculty theorized that students who are that are already good readers might not notice the difference, but especially especially students who are not a skilled or who jumping into academic reading at a different time saw an improvement so. Anyway, just note to all of you out there, this might, we thought this would be an interesting place to dig in and do more research. So, that was just a big, a big jump to us. I think we were here exciting. So a little bit of review about the fact is the focus faculty results with the focus groups and the surveys. We asked in the fall survey what impact did you see on student learning and engagement as a result of using hypothesis instructors said these things were better. Engagement was since with instructors was better engagement with students was better students were better prepared for discussions. There were some improved grades, and there was better preparation for writing. Again, these were 17 responses. There is a caveat here and an asterisk I just want you to notice my asterisk. This reflected what we heard in focus groups, and what was reported on the surveys this was not done with any research comparing grades before and after or same sections or that type of thing. So this is definitely a space where I'd love to see some more research and I know Rami I think Rami clear is doing some of that in Indiana and there's some other places so I'm looking forward to seeing some of that work. But we were excited to see this. Some of the follow up question we had for instructors, again based on what we heard in the focus groups. We asked them. We gave them these statements and we asked how much they agreed or disagreed with these statements and for the most part these, they felt that students were better able to break down and comprehend text. They felt that instructors could see where students were struggling. They felt that the instructors understood student learning at a deeper level and different insight into student what students were learning and what they were struggling with, and many of them felt that students had better questions about the content. I love this quote. This was really an impactful quote for both Karen and I when we heard this, we had instructor tell us that she said it is so seldom that we get to hear every voice in our classroom, the fact that I have three ideas for everybody is This was an instructor who just had students annotate. It's like a three or four page article. She required them to each contribute about three ideas and she said it's the first time in a class that she had heard that much from all of her students, and it really changed how she was teaching. This kind of same similar to this faculty member said seeing the same thing from different perspectives that came out that came out rather quickly did not just facilitate that in class. Normally you'd have to use a lot of class time related to that several instructors said that basically you started class already happened through the material you didn't have to do that background review and you could just jump into the discussions. And the comment that sort of reflects this multiple perspectives and this is something a caution I want to take out. And this is one of the things I want to refer to the book annotation on is sort of this concept of power and where that fits in here and this was definitely something we talked to instructors about this student commented. So a pressure on me to respond with the right thing about a certain topic. So even though you're getting multiple perspectives are students where we think students are feeling very comfortable sharing be very conscious of how students might not feel comfortable sharing and feel like they have to go along with the flow. Her example was sharing about abortion and people having their own opinions. So it was just something really to be careful about. Then this one really jumped out at us. Are you know this is me still for a little bit. Okay, grading annotations. We asked students only in the fall of 21. So this was just one of the surveys. We asked them if they if their annotations were being graded or getting points if they were earning points participation points whatever for this. And 97% said that they were sometimes are always given points. It was interesting that 95% thought it was at least slightly useful to have some sort of a motivation or a point reward on this. The faculty response was a little bit different. That I think they weren't all doing that but I think it's important to see that students really felt that that was important and a good motivator. We asked instructors how often are you responding to annotations and you can see here, most of them said almost always most of the time, sometimes and almost never. When we did ask students that directly, I think it was more on the sometimes and almost never and so student perception and instructor and perception might be a little different. Don't know exactly on that. You can't go into any courses and measure or see what the reactions were again that's a great space to jump and do some more research in in the future. We have students if they would like to use hypothesis in the future. Most students you can see most of them were yes. I want to point out this one it's really hard to see I know just a screenshot from our tableau of the data depends on how it is used. We see how high that number is not Karen I don't think we have a percentage on that, but a fairly significant percentage of students said yes they would use it but it depends on how it was used and we'll talk about that address that in just a minute. One of the ways we look at how it is used. And this is a quote that Karen and I keep coming back to these two quotes and these opposing opinions about student feeling and thoughts about using hypothesis, kind of how it's used, and its presence in the classroom. We have our social butterfly on the one hand, who says it's the only way I can focus on the reading because there's constant annotations to see how my classmates are feeling about the reading, or when the professor. The professor adds a note to further clarify that we heard quite a bit that type of thing. And then on the other hand that every time Karen and I need a laugh and go read this quote from the DIY students a student you just want to do it themselves. I don't get the point. I can read just fine without other people. The teacher should either use it for some purpose or skip it. There's the students there's two sides of the story some students really love it and have that piece and other students don't care so that lends us to that brings us to the point of structures need to be very purposeful and how they're using it and why. Karen I think it's you. Yes, Karen's going to finish it up here. Karen you're muted. Thank you. Instructor feedback. In summary, this is what we heard accountability. Hypothesis definitely helped with accountability and discussion preparation. And we would say accountability accountability accountability but also, I think motivation was just as much a part of the equation there. Because students liked revisiting comments and areas of reading students like hearing what other students thought what their instructors thought and that really helped them reread and be engaged with the materials. Additionally, we saw and heard from the faculty and and student comments that students were learning from each other. Not just about the content but also how to read and critical thinking and some of these skills. So that was encouraging to learn about students. We heard a lot of students are a lot of instructors providing guided readings and direct proximity to text and that was very valuable. It motivated and encouraged students to do the reading and respond. And faculty use type of this as a promising strategy for deeper learning and engaged class discussions this came out quite strongly in our faculty faculty focus groups and that how good the in class discussions became as a result of having discussions prior to class and as a result of course of reading. And Shana, do you want to take these next few. Yes, why don't you just keep sharing. We did, we learned a lot about increasing reading skills this came out quite a bit in several different areas and I would say one of them just as sort of a story and anecdote was in language learning we had one instructor who taught finish finish language. And he talks quite a bit about how he uses hypothesis to clear so we can use culturally authentic materials you can use finish websites and finish newspapers and all those materials using hypothesis to provide vocabulary support cultural support those types of things so students can read deeper into into the materials. In terms of insight and intervention carries on to the next one we also heard. And we talked about this earlier that faculty felt they had a better window into student understanding about a topic and one story I'm going to share is a faculty member who was teaching a class that discussed a lot of difficult concepts around race and racism, which were new concepts to some of the people who it was being discussed. And she felt that using hypothesis gave students sort of a scaffolded way of approaching and learning that content, and being able to process it in a safer space asynchronously, then they were better prepared to go into discussions in the class, and then they followed up with reaction papers afterwards. And she noticed a progression and how students were responding and processing those difficult concepts. I ran into her just a couple weeks ago totally away from work. And she told me that she will not teach again without hypothesis, because of how it helped deal especially with those difficult concepts. All right, well finally we'd like to share our recommendations for instructors that we serve using the insights from our evaluation. So when we first got the feedback bag we looked at and we had a question about what needs to be improved from students and we had all of these comments and like what's going on we didn't know didn't get a lot of negative feedback but there were a couple cases where the PDFs weren't working so well and students really did not like that because it affected their ability to use the hypothesis to annotate. So make sure your PDFs work we found hypothesis has a great OCR tool, it's a very easy to use it's easier than Adobe. Hypothesis there's support say this three times support success representatives helped us occasionally when we had PDFs that were hard to deal with. But make sure those work. Students having instructors interact with students and hypothesis really leads to their satisfaction satisfaction and their engagement materials. So try to think about interacting. You can create an assignment where students just read and respond to each other but the satisfaction seems to really improve when instructors are engaged in the reading as well. A close follow up referring to annotations in the discussions and in class discussions is something that instructors really appreciate or excuse me students really appreciated they like to to have the discussion move from hypothesis into class. Fourth we heard even though hypothesis is pretty intuitive for most students they still wanted instructors to model how to use that just a brief demonstration in class and then a little bit of support documentation and the instructions goes a long way. Fourth going back to our social butterfly and do it yourself for type opposing opinions. There is a visible not visible button and hypothesis and we will tell instructors how students can use that to hide the annotations if they prefer to leave their annotations without seeing others in the future so that is something we will be mindful of sharing. And then if you're using hypothesis for consecutive assignments, we suggest varying the assignments even just slightly just to make each experience a little bit different and novel. Yeah, varying the structures of assignments or considering use of group work categorizing on the annotations with tags or scaffolding assignments is another idea where you're using annotation to build ideas and then you're summarizing them to post in a paper or in a canvas discussion or synthesizing it in another way can make the assignment very interesting. All right, so those are our final recommendation and what's next well, we want to write up a list of use cases and learning activity suggestions for instructors and to help with come up with some of those new ideas and to make each assignment just a little more interesting. And then Shawna, you have the book you have to hold up the book as a follow up we. Am I saying. You go go Shawna. I'm trying to hold it up in front of me so you can see it. We are Karen and I are going to be reading the book in cooperation with a couple of faculty this summer, and to be able to better incorporate the concepts from the book into evaluation and into development of other pedagogical pedagogical approaches so thank you so much that's our. Our next step. And with that, if others have questions we'd be happy to answer them at this time. Jeremy you're on mute, and also someone wanted to know if you can hold up the book again because they couldn't see it. Can you guys hear me now. Oh I think it's because of your virtual background. Oh yeah, yeah. Yeah, we can hear you Jeremy. Okay, great. Thanks so much for that exhaustive exploration of all the great usage we've seen at the University of Minnesota, Twin Cities and Duluth. A lot of really interesting things going on. There's been some questions that have been asked in the chat. But I wanted to ask a couple questions before we open it up. Sean has been very good at answering the ones in the chat, but I'm really interested in the point that you guys highlighted about the student who said, or rather the number of students who said, yeah, I'd like to use again. If it was, if it was, depending on how it was used, right, and even the student that was negative, like it should be used for a purpose. And just talk a little bit more about like what does purposeful use of social annotation, or hypothesis need and actually maybe we should start off with like, what would a purposeless. Or how does, how does a purposeless usage. Is there a purposeless usage or how does the appearance of a purposeless usage happen. I don't know, maybe you guys can riff on that starting with Shauna. Um, yeah, I have some examples of things that we heard that I don't know if I would say purposeless but I would say could use more thought using it every week for the same things over and over having and I one instructor who fully admitted she didn't do this right as she had them and it can how many was it it was like 2018 it was a huge number of assignments articles just always the same thing over and over again I know one instructor used it this semester and I worked with her to pull it back into put some variety instead of always having them use hypothesis, make it something. You know, specific question or a specific purpose or specific point differently in each reading, because otherwise it gets just redundant and not engaging if they're always doing the same thing. Yeah, I would say that make three comments and two replies model is okay the first time and then the second time you know asking them to specifically look for something or do something. And that's where Karen and I felt that the images adding visual like visualize what you're reading go find a gift or an image that represents that, you know, mix up how you're doing that, you know it's great to use hypothesis for all the weekly readings but yeah don't always do the three comments and two replies. The tags could be helpful there too right in this particular I want to think about these themes so when they come up maybe tag your annotations with those themes so there's a little bit of a new scavenger hunt every single time that you that you annotate but I think that's really interesting the idea of being more purposeful on deliberate in your annotation assignments. On some level I believe annotation sort of a good in and of itself. I think to some extent, you know it is writing in tech writing your book is you know something people have done for a long time to help with comprehension and analysis but but really helping lay bear for students what that means exactly. What is the work that we're doing what different types of work. What are we doing with this tool and making that part of the assignment making that making that part of the instruction and making that part of the sort of development of skills related to annotation. I think would invest it with more obvious purpose. So that was an interesting one. All right. I just popped up here. Maybe this is a comment or question. The other question I want to ask and then we'll open it up in the last 15 minutes here. That student who was worried about saying the right thing. Is that something else that can be avoided with more purposeful use of the tool tool. I think that was a piece of feedback, the reluctance to speak is your or the reluctance to freely speak because it was feeling like you had to say certain thing. I think. And we didn't hear that a lot, but I'm guessing that is more present than we heard about it, especially if you're talking about difficult concepts and it was a different class than the class that was dealing some of the real complex issues around racism. I think that is that can center around hypothesis but it also requires a culture in your class that will make that you're addressing that over and over again and that's one of the reasons we also really recommend that there's an instructor presence. I mean you all know this this is why the instructor presence in that annotations is important because if there is any sort of. Issues that we're bullying if there is any of that or you're seeing sensing that students are afraid to comment or that there's people attacking other students for what they're saying that I think that has to be addressed as a class culture but also a very purposeful assignment in, or, you know, a topic of one of the assignments to be we're looking for controversial things in this assignment and we are not going to be attacking people for saying what they think. That's just up top of my head. Yeah, and also I think we'd be interesting and interested in hearing how people facilitate, you know, those discussions of diverse opinions and how they're making students, you know, helping them not do group think but also feeling comfortable in the space that they're in. So, definitely more research, more learning there. Yeah, I think the point about instructor presence is a really good one. Of course, you know, that's asking more of our already overworked instructors, but I do think that's an important piece of not just moderation and motivation, but I think it actually ties back also into the question of being purposeful. There are instructors there responding, engaging with you in conversation. It's hard to say that it's a useless activity, right, because it's sort of obvious that somebody's listening to you, somebody's responding to you. Hopefully it becomes clear that that's a generative conversation kind of the bone of what we do in academia. All right. Shawna or Karen has there anything in the chat, Shawna you've been very vigilant there. Thank you for that but anything that you think is worth surfacing in the oral space here is. I know you were quick to answer some of the questions but anything you'd like to surface and elaborate on from the chat. I've been distracted by the questions to which is fine. I shouldn't be answering them as I go. I want to address a couple of people have asked about the canvas course and I think I, I'm not going to go distract myself looking for it right now. I'm not sure if that's something that's limited just to University Minnesota logins but that's something I think hypothesis has a lot of those same resources. We just package them into a canvas course because that was something that faculty at the University of Minnesota were used to using. Honestly, most of it was much of it was pilfered from hypothesis. Yeah, it's our, it's our basic practice when we're able to for any school that we're partnering with to set up a course that's for instructors and instructional designers. That has resources and can be a kind of hub for a pilot for an ongoing subscription to the tool, but also a space to play. So whenever we can, we'll set that up in your LMS instance. But of course those are restricted by logging into the to the instance. So yeah, I doubt that people can get into Minnesota one. We supplemented it with specific examples from the colleges courses, we had a couple videos in there that were doing some specific trainings that I think we took out because things changed. I'm no longer managing it so I don't know what else they've done to it. Veronica asked about automatic grading in the chat. One of you guys want to answer if there's automatic grading available. Here. Well, it's certainly easier to grade with the canvas integration and speed grader so there's that. There is not automatic rating. However, I do work with one instructor that uses a rubric that allows you to grade. And so she sets the criteria and then she goes through and, you know, clicks where they fall in the rubric and that expedites her grading quite a bit. So, is that something that she could share, or she'd be willing to share. Yes, or I could simply. I mean you can set up the LTI or the hypothesis learning tool on canvas and then there's also a rubric so if you create an assignment in campus, you can click the plus rubric button and I think if you do that before you install hypothesis, then the rubric will be in the assignment. But I think you have to do it before you install the hypothesis in the assignment. Thanks. So the answer, the answer to Veronica's question is no, there's no automatic grade with hypothesis. There are some other platform reading platforms that do have kind of algorithmic grading. But what Karen's been describing is we have a pretty deep integration with the grade book functionality in the LMS is an especially with canvas with their speed grader and you can add a rubric. But our model is at least for now, and our focus is on, you know, making the student work visible to the teacher and allowing them to design the rubric or algorithm, if you will, for for their evaluation and feedback to students rather than fully automating that any body in the audience want to raise a hand is that possible in this context. There's a question from rainy that he wanted to know that guy. I know right. He wanted to know what what disciplines were covered in this work. We did not mention that in the beginning but we did not mention that in the beginning so thank you all right Karen we never even that never occurred to us to mention that idea. Yeah. So I did put a couple in the chat this but let me just write I've got our list up here, everything from course on introduction to Buddhism through our Asian and Middle Eastern studies. And education course on assessment of secondary students, several English courses. I know our first year writing seminar is using it now that they were not part of this evaluation, several language courses finished German Spanish journalism, linguistics, many history courses, which is very new and dear to my heart. Lots of language Russian. We have philosophy using it here, several language Spanish, I think predominantly here French writing and education, I think we have two or three instructors and education using it as well. Wow, that's, that's quite the breadth of disciplines that's awesome. Any other questions in the chat. I'm just going to address Samantha's question about stem on the Twin Cities campus it was used only in the College of Liberal Arts which most of the stem departments at the University of Minnesota often like the College of Science and Engineering or the College of biological sciences so that's why my the ones from the Twin Cities campus tend to lean to lean humanities type courses. Can I just quick talk about Alan's comment and then I'm going to stop talking. Getting students to return to an annotation for discussion that is definitely an issue that came up from instructors was you know the first students would go out and make a comment and then getting people to come back, which is one of the reasons, one of the instructors had students write reaction papers based on those annotations and that was a great way to get students to come back. But that is a comment and heard quite a bit. We have another instructor that requires, and she like just makes them separate assignments in canvas. So it's kind of like a prompt in canvas to return back to the conversation but then she also uses a Google form and has, you know after the students spend a bunch of time conversing in the margins. And then they reformulate their questions to submit before class. And so that really gets them thinking about their questions again and preparing for class and they get some points for submitting the questions. That's one strategy that seems to work well. I'm going to run to this question about the video. It wasn't annotating video and multimedia. It's using video and images in annotating written documents because I, as far as I know, the hypothesis is an annotating video. So, absolutely include in the way they have annotated video is by using the transcripts and I know of some people that have done that, but we're talking about having having people having students add visualizations and using visualizations to reflect what their reading is. And there's been some, there's some great stuff when that happens. One of the things that that I'm thinking about as we wrap up here is just the, the, the work we've done over the years, right there's clearly a value to your faculty in this tool, but one of the neat things about somewhat new technology that really adopters like the University of Minnesota is learning about potentials we haven't explored collaborating together on going and, and, and, you know, implementing both things like adding more multimedia so I think one of the really great things about the partnership that we have University of Minnesota, Duluth and CLA which just renewed their contract yesterday. Sean, I don't know if you know that. I did not and I'm thrilled. Good. Yeah. So, totally done deal for next year is that we can now, and I know Shawna you moved on, we're connecting with some other folks you know that in your former office and, and Karen you're still there is that we can work together to expand and deepen the practice of social annotation at these college at these universities and campuses, based on, you know, the really reflective work that you guys have done with the initial cohorts so thank you very much for all that work. I guess now I just had a question in the middle of my sort of closing statement but the faculty focus group was something I'm not sure that I've seen at other pilot institutions and can you just talk a little bit more about how that was run. Because I think it could be a cool model for other schools that are really being deliberate in their exploration and adoption of technology not just of social annotation, in terms of really getting high quality feedback from those early adopters. So you want to run with that one. The faculty focus groups so we ran those in spring. At the end of spring and we had three sessions, and a list of questions to run them through and I think we have five. They were virtual sessions so we had five faculty, approximately in each of the three sessions and we just ran them through the questions and we recorded the sessions just for some news and took notes and then reviewed the notes and came up with a new list of questions then to put in a follow up survey. So that I mean that's a kind of a higher overview. It was just wonderful to have them all together and you know we had everybody taking turns as asking and answering questions but putting them all together in a room where they can kind of reflect together there was a lot of yes and statements that help us and cover some of the aspects of use that we were wanting to know. Faculty really enjoyed the opportunity to get to talk together about things, and it was great because Karen and I just stayed out of the way and let them converse. We had thought and we never did this was create sort of a community of practice and maybe CLA I'll pick this up now next year is creating an annotation community of practice which because faculty really, we did have a couple of those where we had conversations. And it was great to see people getting ideas from each other sharing solutions, sharing problems and how they fix things and ideas it was really, it was really, really good. Hey, I'm just going to jump in to say we're right up again done up by the hour that went by so quickly. I can't believe when I looked at the clock. Yeah, any closing remarks from anybody before we wrap this up. And also thank you for staying on a little later and apologies to, you know, we hadn't quite decided if we were going to do that so normally I would have advertised that to people. Thanks all for listening to us and letting us share this experience I'm sure it's all things all of you already know so. Yes. Yeah, thank you. Thank you for the continued collaboration. Yep, likewise. Thank you. Thank you. And I just want to, yeah, thank you to our wonderful guests and to Jeremy. And Aaron and also I think she had to leave but Becky who was holding down the chat. We have such a great group today so we will see you next time on liquid margins. Take care everyone.