 How do you know if you're achieving the goal? How do you know if your students are meeting the mark? What do you do? Again, we kind of moved very quickly from a lot of my classes, which were many paper-based quizzes, tests, final exams, to all remote assessments. And it was difficult. I mean, I experimented with some variety of types of assessments. I tried some group assessments. And some students liked them. And some said, no, I don't have time to work in a group. I'm working full-time. I'm not on the same schedule as some of these other people. And there was some pushback to the group assessment from some. And there is this concern in universities that if we're not doing a proctored paper assessment, then students are cheating. And we can't trust anything. I really couldn't focus on that when we were completely online because I really didn't see what the alternative was. So I did do lots of remote assessments. I did lower-weight assessments. I didn't do a final exam, which was kind of new for one of these first-year courses. I did term tests and frequent weekly sort of mini assignments and had students sign a academic integrity statement and hope that that was enough. I know I'm getting into a whole other ballpark here. But yeah, lots of low-weight assessments was what I was looking at to try to see how students were doing. There were many themes there in that response that I want to dig into. So there's these concepts of mastery versus getting a good grade. And as an instructor and in this role as an entrepreneur, I would love for technology to assist instructors in bringing students forward with mastery. And I'm less interested in just taking the box so that somebody can pass the class without necessarily really achieving knowledge the way that we are supposed to be imparting. So I'm really interested in that kind of theme It's one thing to just hand out grades and have the registrar be happy that there was a proctor involved with the exam. But that's not the same thing as really helping my students that I'm newly empathetic with to achieve certain goals. Jana, what do you think about this thread? And I definitely had the usual issues of how do we suddenly transfer things to online? It definitely has led to a common theme of reducing the big high stakes assessments. My course I'm teaching right now is actually the first one where I've done it without tests. Because in an online environment, I'm not sure what the difference is between an assignment and a test apart from, for one, I'm trying to mimic what I would do in class. It has the benefit of there'd be oftentimes in my stats classes where I would be teaching something like normal tables. And I'd have to teach them a way to do it on a piece of paper because that's how I would have them write their tests. And then the realization, well, why am I teaching them how to find a number by looking it up on a piece of paper that I photocopied for everyone? I wouldn't do that for anything in math class. And finally, well, if we're doing it all online, I can just let them use our and write things in the notebook and they have the code available to them and they could fix it live while they're writing their tests. But it took a surprising, on my part, a surprising amount of time to accept that we're still talking about the normal distribution. You're still learning the same amount of stuff. You just haven't learned the esoteric skill of looking up numbers in a weird way along a table. But it seemed like that was, oh, we're cutting that out. But really, it's the typing it into a program that makes a lot more sense. I'm not going to make them learn arctan by pulling out a book off of the shelf and looking up numbers there. But it took a big, a lot of time for me to accept that I'm still teaching this just in a slightly different way. And having that, the assessment where they do get some of the feedback of, okay, you need to do this. You can at least see if the computer is doing what you think it is before you send it off to me to get some feedback from me is so much nicer than, okay, you wrote it on a piece of paper. You've made a silly mistake, but you couldn't possibly have known it until I've graded it and told you this was wrong. You give them the chance to fix their mistake because that's how you do math, right? The fixing your mistake, the understanding, something went wrong and let me figure it out rather than, oh, I made a mistake. I lost three marks. So that last bit is really interesting. I want to follow up with you immediately. So dialogue, the peer review process for the researcher is analogous in some sense to providing constructive feedback to students so that they can process what they did wrong and then improve. Did you engage in more or less dialogue, Janna, during the pandemic? I tried to do more. There was always the time constraints that we're all working with. I taught a proof class in the past term and I was lucky we had access to hypothesis which allows for social annotation. So that was really nice. It was, we were doing some synchronous classes but it meant that there was a process to check in with the students who were asking questions about things in the textbook. And so I found definitely with online teaching, one thing I need to work into is, okay, I need to be checking in with them every day in some way so that there's some kind of, there is more interaction. It's no longer, okay, I teach 50 minutes, three times a week and then answer some emails. I need to have more of that feedback. And I just sent my current students, they handed in their worksheet, their first playing around with an hour notebook and sent it back to them this morning with the comments of, you know, some of them were just, yeah, that looks good and some comments about the data set we were working with and others that though, here's what you needed to type in to make it work. That's interesting. So you're using hypothesis to provide feedback back around our notebooks, for example. I was using that for the textbook and I'm using CrowdMark for the air notebooks. I get them to submit it and play around with it. Very cool. What about you, Sean? I think in the past I had the sense that you have some strong opinions about grades versus mastery, so I'd like you to throw some spice into this conversation. Yeah, I could keep going on this until our time is up, I think. I'm moving in the direction now where I'm no longer convinced that the, you know, the timed proctored closed book test is this sort of gold standard for assessment. So I was teaching what, for me, are small classes of about 50 students. With 50, I said, hey, these classes are small enough. I can, maybe I'm not gonna go full on, you know, like mastery grading, but I can at least let them do revisions on like any piece of work they submit. They can have at least one chance for revisions. And the revisions are nice because, you know, now some of them will just email them to you, but others will book office hours and they'll come in and then you get to have a conversation about what they know and what they don't know and they can kind of work out and get to the point where they're like, okay, now I understand what I need to do on this problem. I can go away and I can fix it and I'm gonna send you the improved version later in the week. And then you, you know, when you start getting the feedback from students, you know, I got a lot of comments from students saying, like, oh, wow, like, you know, I appreciate this because now I can look at tests as actually an opportunity to learn something and not just as a measurement tool, right? And that really changes the conversation. It really changes the perspective on what they're doing, you know, in that test. And you've removed some of the pressure because they know they'll have another crack at it if they get it wrong. You've removed some of the incentive to cheat. I mean, and there will be, there will still be some, even though they have like unlimited attempts to get it right. And it's just that sometimes the students get to the point where like, I need to have this done and I don't have time to do it because I have 1,000 other pressures in my life right now. So I'm gonna let somebody else do the work for me, you know, and that doesn't make it right. But at least, you know, those of us who are seeing, you know, the cheating going on in our bigger classes, I felt a little bit, you know, I don't know, vindicated that, you know, even like these expert teachers teaching 20 students at a small liberal arts college are seeing the same problems that I'm seeing.