 We're excited to be here with you guys today just to give you an intro while a few more folks join in I'm Suzanne Miller. I'm a customer success manager with hypothesis. I already mentioned joining you from just outside Raleigh, North Carolina. And I am joined by my colleague Chris. I'll let her introduce herself. Hi everyone. My name is Christy. I, like I said, I'm in Southern New Jersey kind of outside of Philly. And I am happy to be here. I have been using hypothesis in my own teaching since 2019. But I also am a customer success manager here at hypothesis. Now, we have an agenda here today a rough agenda. It's going to go pretty quick, but we have plenty of time for questions. And we've got a few folks helping us with the chat joined by our team I think almost entire team is here today as panelists so that's pretty awesome. But wanted to take a moment to have you guys. I don't know where that came from. Have you introduced yourselves in the chat. Since we've introduced ourselves. So if you can make sure you pull down in the chat, the blue button pull down to everyone and share a little bit about your, your school, obviously, and your role, your department that sort of stuff. And even your experience level with hypothesis if it's brand new to you, or if you are a seasoned veteran. And while y'all are introducing yourselves and also going to launch a poll just to see what LMS you're using, because this will come in handy a little bit later on in the session. So if you could also take the poll. Sorry, we're asking you to do many things at once. Not for a Friday huh. Negative 30. Okay, I think most people are taking the poll so thank you for letting us know. So we can take that into account a little bit later on. Absolutely. So, I am going to plug on sort of set some purpose here with our session today. This actually comes from a blog published in 2019 by several instructors and including our scholar in residence Remy clear. But it had some will call golden nuggets of information about grading and feedback. Obviously, it was a study to talk about motivations and engagement with social annotation. But I think the key in this blurb we have here is that participation marks or the fact that annotation was required for a course was by far the biggest motivation for students to annotate using hypothesis. So it's something we cannot underestimate and, you know, while sometimes it can get, it can bog us down a little bit as instructors. It's obviously a really important part of this practice that you're using with hypothesis in your LMS. Was there anything that you wanted to add Chrissy about this. I know speaking from experience. Okay. Okay, so again, we're going to this is not a poll but this is more us sort of prompting you to get some ideas about what kinds of things you want to see from your students in their annotation so like in a perfect world. What would be some of the things that you would love to see and feel free. If you don't see it on this list to go off script. But if you see a combination of things you can share that as well but if you can in our chat, looking at this list, share with us what are those things that you are looking for in your student annotations. Thanks, Bridget and carry it. I'm seeing I'm seeing a trend here for sure. On the analysis the critical thinking reading strategies. Very good. I dare say that might be why you're here today. I really want to talk about those high quality annotations because that's where the grading and feedback really plays a major role. Okay, insight. Okay, very good. So some of it is about, you know, quantity, as far as how many annotations but it looks like a lot of folks. It's about how your students think through the reading and thinking deeply through that reading. So hopefully today that sets a little purpose because we can talk about how you can dig in a little further and assess and give feedback on those deeper annotations so thank you so much for sharing everybody in the chat. So the type of grading that we're going to talk about is analytic grading. This is where we're dealing with learning objectives they are the obviously the essentials for your course or for that assignment. Professors obviously construct rubrics that represented a level of achievement for each of those objectives or elements. It does not yield a single holistic grade but it does reveal its parts. So it really allows you analytic scoring to compliment if you are asking for analysis for your students right. So for those of you who shared about analysis being a, you know, a heavy component. I think the analytic grading scale could probably be your best bet moving forward. So that is what an analytic rubric would look like. I've actually in this slide shared an example from two instructors at the Missouri University of Science and Technology. We have credited them here and you guys actually have the links to the slide so you can come in here and dig a little bit deeper but this is, you know, fairly involved analytic rubric. As you can see here, we've got each of our objectives, our elements that the instructors were looking for and then we've got a scale that follows from there. So in the ways that I mentioned earlier it does reveal each element, each part, and you can really look at strengths and weaknesses. Obviously some of the advantages of analytic grading is that it brings to consciousness. Conscious processes that go into recording grades. So, you know, things we always talk about hypothesis making learning and reading more visible and analytic grading rubric can do that. It looks at those strengths and the weaknesses, and all the criteria and are weighted to reflect the relative importance of each element. I don't know if you saw in the last one but certain elements were worth more than others. So it can be much more objective, which is nice of course, and it does allow you to give more specific feedback, which will lead into our feedback part of the session a little bit later on. And with that said, we're going to talk a little bit or Christie is going to share with you about holistic grading. So, I think it's really important to keep in mind as we talk about each of these different methods or strategies for grading. What is the actual purpose of your annotation assignments and a lot of you shared kind of a general purpose with us. The analytic rubric, I think is really great. Just as an example for something like if you're having students read academic articles and you really want them to identify like key parts and you're asking them to annotate certain areas. There's really like a breakdown of different things that they're doing. Just one example could obviously be used for other things as well. The holistic grading rubrics are a little bit different in that to state the obvious. They're more holistic. They consist of just a single scale with all of the criteria for the evaluation in one kind of grading area. So the students are kind of just you choose a box that you're going to be assigning the student to describe the kind of their whole annotation assignment. So this is one example of a holistic annotation rubric. This might be good for something or someone who is using annotations in a more general way in their course. Like if maybe you are having students complete different types of readings and you kind of just want to go through their annotations a little bit more quickly. Instead of breaking down like these different pieces that you want to look at. Next slide please. So the holistic grading advantages is that we're looking at overall what the learner is able to demonstrate instead of kind of picking out and rating low what they're not able to complete. It also is a little bit faster to complete your grading since you're only looking like you're choosing an overall description for how the student has scored. And it can be applied a bit more consistently if you have trained raters doing the grading since there's fewer options. And then that can increase the integrator reliability. And then finally I want to wrap up with a another rubric option known as a single point rubric. Basically I like to think of this more as like a checklist. So it's kind of like an analytic rubric in that it's listing different criteria of what the students should include in an annotation assignment. But instead of describing what a deficient annotation might look like, it only is describing what proficient work looks like. So only proficiency is described. It's only describing like you should your your annotation should have X, Y and Z. You should have three annotations. They demonstrate critical thinking, et cetera, et cetera. So they can be pretty quick to develop since you're only coming up with that description of what a proficient annotation is. But it's a little bit limited in that the students don't get a good sense of, you know, what maybe they have not done well without further comments since that like deficiency description isn't provided. But we have provided in the slide app we saw examples from faculty at different universities for analytic and holistic rubrics. But we also have linked to just kind of slightly simpler versions here that you can get started and adapt for use in your own courses. So if you want to check out some other examples of generic analytic rubrics holistic rubrics and a single point rubrics for social annotation. You can find the examples here. And I am going to pass it back to Suzanne now. Yeah, thank you Chrissy so yeah slide 13 is going to be super handy, because like Chrissy said these are these are great starting point, especially if you're like dabbling with one or the other. I'm going to talk a little bit about the art of feedback. And I don't know, I think Chrissy might, are you going to be hopefully launching a poll for us here before I go or should I move on and maybe the poll will be later. Yeah, I can launch that. So if you could just let us know. Actually, Suzanne, do you mind going back one slide just so we can remember which one looks like which which rubrics do you think you might prefer to use in your course with social annotation. Oh, we're pretty split. I know. We're actually, unless other, oh no, no, it's really funny to watch it you get y'all can't see both rolling in but it's like the different rubric types are battling it out. Yeah. Yeah. So far holistic is sort of edging us out with maybe one additional person. Yeah. All right, I'll share those results out while Suzanne continues so y'all can see. Thank you guys for sharing that. So now that we get an idea of maybe the type of scoring that you want to do. I'm going to talk a little bit about how you can provide that high quality feedback because you'll notice that even while we were talking about each type limitations and advantages feedback was always a component. I really loved this particular quote from Thomas Gussie Gussky, specifically the idea of the fact that students really don't get any kind of direction for improvement. You know, grade assigned grade does not equal learning right. And so only when that learning is paired with individual comments offering some guidance and direction for improvement. That learning really happened and enhance achievement. So the idea here being that obviously social annotation offers an avenue to provide public feedback. But maybe it's not so specific to the quality and improvement of the annotations themselves. So with that said, let's just talk a little bit about it. Obviously social annotations and instructor feedback can model very much like a studio or workshop approach. Obviously it helps students sort of self regulate take the initiative they need. And obviously in that role, the teacher is to provide that guidance and that feedback as a facilitator. But I would say in the margins of your hypothesis assignments, it almost operates like an art studio or gallery where the work is on display. And it's shared openly for that group feedback. So we don't want to underestimate beyond grades, the importance of the reply function in the annotation or in the sidebar because obviously that reply. While it is group specific can really prompt contributors to really up their game, so to speak, when it comes to sharing annotations. As far as critical feedback, which we know would probably be a little more private between the instructor and the student. I would say the emphasis should be on quality over quantity. You want to focus feedback on primary traits or those learning objectives that we talked about earlier. And you really want to use as much language from the rubric as you can. I think that accomplishes two things. Obviously, it's very specific and, you know, constructive for students, but I also think that it reinforces using the rubric and helps students makes them want to return to it throughout your course or throughout multiple assignments. But I do believe also that it, you want to make more recommendations for how a student can improve rather than identifying those errors or weaknesses. Some people might say, well, perhaps the single point or the holistic rubric would help me in this effort to, you know, focus on the positive and focus about how, you know, how they can improve and that sort of thing. There are four principles of effective feedback. Obviously, you want to emphasize the task. You don't want to dwell on the students ability, because that can get complicated. You want to give them that specific feedback so they can improve. You want it to be regular and ongoing. So I would dare say if you're enabling multiple hypothesis assignments over the semester, which we hope you are doing that you are staying consistent with the feedback that you might be providing through usually for your LMS the grading feed, you know, great great book function is where you would provide that private feedback, or maybe on a standalone outside of the LMS but however you're providing it more privately, you want that to be regular and ongoing. And more so than anything I think nowadays it's probably most important that you focus on the process. But the results that's one of the reasons I love hypothesis is that it's a very process learning process oriented tool, as opposed to that that end product, which, you know, doesn't provide opportunities for ongoing feedback. We've created a an acronym here that might come in handy we call it art. And what art stands for is what you see on this slide. And this is to affirm sharing of course, you know, what's good here in the annotations that you provided. And then again, this is much more in a private back and forth that you might have with the students so sharing those strengths reminding them to not do it to add this or do this. If that was included in the assignment and then of course, kind of giving them a task, like a next step what is it that you need to do now to move forward, and to improve upon this assignment. With the process of art, we decided that it would be cool to actually take some feedback from a couple samples and see what you could do based on what you've learned today and of course your wealth of knowledge over time. Share with us in the feedback jam that Chrissy is just posted in the chat. What you're going to do is you'll notice that there are two examples. We'll say page one and page two of the jam. And what you're going to do is you're going to pick one example that you like, or that you think would challenge you. And you are going to use the sticky note function, which could be fine on the left side of the jam board. I've circled it on my slide if you can see it. And you click that and what I'd love for you to do is you're going to see the teacher feedback that was provided it'll be circled, but I want you to try to use two of the art elements. To provide or improve the feedback or improve upon the feedback that was provided from the teacher on these assignments. And with that said, I am going to go back a slide so that you can remember what those elements of artwork. I'm just making these a little bigger. If you happen to be looking at my screen so you can see it. Awesome. It looks like a majority of our folks probably have added here. So, awesome. Hopefully you're taking a time if you if you worked on one. Take a look at the other so you can sort of glean some new ideas some other ideas as well. Is there anything that Christie that you would add do you think we've got a majority or. Yeah, I think we kind of. Move on. Okay, awesome. And we'll talk a little bit about some of your takeaways from the activity, but I'm going to actually stop sharing so that Christie can get more into the nuts and bolts of hypothesis and how you can use what you've learned today and hypothesis. I just want to give a quick review of what it actually looks like to grade using hypothesis in a couple of different LMS is. So we talked about different approaches to grading today we reviewed holistic using holistic and analytic rubric so you can use those in your grading. So I'm going to give the student the basic scoring guide like with the single point rubric, and then I also did just want to point out here that if you don't want to use a rubric at all, you can always just give students a complete or incomplete grade, or a past fail. So actually in my class because I really rely on social annotation as a way for students to engage and participate in the class. So basically all of my annotations are graded as complete and complete, like you either get the points or you don't get the points and I give more specific targeted feedback to the student to kind of try and guide them in their future annotations. So just some thoughts of how you could think about assigning grades to annotations. So most of the participants here are using canvas so I'm going to go over both how to grade in canvas and then how to grade in non canvas LMS is because canvas uses its own system and then all of the other learning management systems that are not canvas looks the same. So in canvas hypothesis actually works with speed grader. So I'm just going to open up one of my hypothesis assignments in speed grader. Sorry, I thought I had this open already. I'm going to open my speed grader in canvas, and you'll see that in speed grader hypothesis will filter out the annotations by students so I'm looking at Jennifer's annotations here, and I'm only seeing Jennifer's annotations in the grading sidebar. So I can just review what Jennifer has contributed, and then I can give Jennifer that grade if I want to end the comments that private feedback and move on to the next student. So I want to highlight and this will be the same no matter which learning management system that you're doing that when hypothesis filters by student, sometimes professors are like okay but I want to see like what they're replying to like if they're if they're replying to another student like we see with this example. I'm looking at Malika's annotations and she's actually replying to Jennifer in one of these examples. So you do have the option to expand and kind of show that context if you want to. So while you're grading, you can reveal more context or choose to only look at, you know, the one student at a time. So that's what we look like in canvas and this is the slide is linked to our instructions for how to use speed grader in canvas. I'm not going to go through the steps here I just want to note that you can also attach rubrics to your hypothesis enabled readings in canvas. There are just a specific order you have to do that in. If you're interested in doing that. I would make sure to review the instructions here or review it with your customer success manager. In other learning management systems the grading looks a little bit different than canvas so I know we have some blackboard users in here and a Moodle user. I have this open in blackboard but it will actually look pretty much the same whether we're in blackboard in Moodle in Sakai, you know whatever other learning management system. In this case, I have my annotation and text on the bottom of the screen, and I have my grading bar at the top of the screen. So I have this drop down here that shows me any student that has opened the document. So people get a little bit confused by this sometimes. Your student will show up in the drop down menu once they have opened the document, even if they haven't added an annotation yet. But if they haven't actually gotten into the document they won't show up here at all. So I have two students that have opened this document, I can filter to one of them. So I can see here again that if I look at Enrique's annotations and I'm looking at Enrique in the grading bar and it's filtering Enrique's annotations on the hypothesis sidebar here. Similar to what I looked at in Canvas, I can expand that context if I want to see who a student is replying to. After I review the annotations, I can enter the score into the grade bar at the top and submit the grade before I move on to the next student. And again, the annotations will be filtered. So the arrangement of things is a little bit different than the Canvas speed grader, but we have pretty similar functionality going to look the same in Blackboard, Moodle, Sakai, our other learning management systems. One really important thing to note about the grade bar in these Blackboard, Moodle, these other non-Cambus LMSs is that no matter what, the grade bar is always going to say that your grade is out of 10. So in this instance, I actually set up my assignment to be out of 100 points, but the grade bar will still show out of 10 no matter what. So you just need to know that you can set up your assignment to be worth something else. So if I wanted my assignment to be worth five points, I could put that into the assignment settings when I set that up. When I put the grade in here out of 10, it will scale my grade back to the grade book. So if I wanted to give Jamie here an 8 out of 10, an 80%, that would scale back to a 4 out of 5 in the grade book unless my brain has forgotten math, and I'm doing that wrong at the end of the week. But it should scale back to your grades. So that is an important thing to kind of be aware of so that you're grading out of the correct amount. And you would say just to add in there as far as in the grade book is typically where you have an option to add comments. Like, if not in the hypothesis grader, but for those other LMSs, is that true? Yes. So if you wanted to give private feedback to the students, you would go into the grade book itself, and that'll probably look a little bit different depending on if you're in Moodle, Brightspace, Blackboard, whatever LMS you're doing. You can give private comments in the grade book itself. So this is what I had just noted about grading out of 10. And then we're going to wrap up with just a couple of things that Suzanne will finish us up with. Yeah, thanks Christy. Well, this is just mostly an idea of what you have taken away from the feedback activity and then obviously some of the other things that we shared today so you do not have to answer on all of these three you just pick. Maybe one that was your biggest takeaway and add that most relevant question or thought in the chat. We'd love to get your feedback. And hopefully you guys, we have links to this, the slide deck so and obviously we also have links to our knowledge base, especially on several of the slides that Christy shared so if you really want to dig in deeper on what the grading looks like in your LMS. Well, thank you is there I think we've got some resource slides here for you, which we always share. We've got our liquid margins we've got a very hot topic one coming up. I think there might be some links to registration on this slide about chat GPT. And actually being a good compliment to what we talked about today. When we think about chat GPT and the implications. And again, it makes me come back to the idea that I love that hypothesis as a tool is much more focused on the process, and not that end product. We might see more folks potentially leveraging hypothesis and their courses for that reason. We have tons of great resources for social annotation assignment ideas so definitely check that link out. We have a hypothesis educator forum which I think a few of you guys might be in. But if you click that link you get instructions to join that and really just have people to bounce ideas off of get support from and gather even more ideas than what we have shared with you today so really kind of crowd sourcing your hypothesis support is awesome. And then of course we've got our technical support if you ever need to reach out with technical issues you might be having with the tool. And then, I don't know do you want to talk a little bit about hypothesis Academy Chrissy and I'm putting her on the spot but. So we have a new kind of asynchronous online option if you want to learn more about hypothesis and practice developing your own assignment. We have a hypothesis certified educator credential which is a two week asynchronous course where you collaborate with faculty from across the country and work to see, you know what people's ideas for using social annotation are in their courses. And our next cohort is launching March 14 so there is a link on the slide that you can where you can register. And if your school is a hypothesis partner. So if you have a subscription to hypothesis if you can use it in your courses then your hypothesis Academy registration is included in that there is not an extra cost. I think the link should work Bridget says she's having some issues with the jam link or I just clicked it worked but I want to make sure. So partner workshop links your inner partner workshop right now if you're interested in joining a future one they, they're available here so you can check out upcoming dates and register for those. I think, if you have any questions sorry I just took, I just took over Suzanne. It was like the developer, and let our first cohort so I was like, I'm not going to attempt to talk about the academy then Christie since she's here so thank you. So I just want to say if anyone has any, I know some people are new to hypothesis here if you want any more support on like setting up hypothesis assignments. How do you use it in your LMS want to chat with someone about just like how to best use it in your course email the address on the screen here and we'll get you connected with the best customer success manager, who works with your school, and that's, that's about. I think that's all we have for today so if we have any other questions, Bridget I was actually clicking on the link from the chat, so I wouldn't try clicking on the link on this on the zoom screen I was clicking on the link from the zoom chat itself. And that was working for me. But I will be sending out the slides via email as well. I'm going to get that recording to so lots of resources and lots of ways to follow up. Thank you guys so much. Obviously have a wonderful and warm weekend as much as you can. We appreciate you.