 I'm Jason Harlow. I am a senior lecturer in the physics department at University of Toronto. I teach a course called Physics of Everyday Life. It's a science course meant for non-science students. It has about 350 students in it, some from humanities, some from commerce. And I have two midterms in the course, and I crad marked both of them. And I have eight graduate student TAs who have to share the marking across all these students. So being able to scan the tests and have the TAs mark from their offices and each TA mark a particular question to keep things fair seem to streamline the process quite a lot. So the ability to add comments was very useful. So I like students to get feedback about what they've written and particularly where they've lost marks. And often the same mistakes come up again and again. And normally when you're marking by hand, you're writing out the same comment again and again and again. But what my TAs found themselves doing is they kept a word file open where they had pre-typed out the standard five comments that keep coming up again and again, and they just cut and paste them right into crowd mark. And that way giving students good feedback and also speeding up that process. So that was good. Also, I guess crowd mark has these features where you can use keyboard shortcuts to go through. Because a lot of my questions are either 100% or 0%, not a lot, but maybe one in five or something. And so just being able to say, okay, 10 out of 10, just type 10J, it goes right to the next exam very quickly. And that helps speed things up. Which by the way, when you compare it with grabbing from top from the stack, finding page three, and then writing 10 and then writing on the front, then putting it in that stack over there, it's a lot faster. In the remarking phase, I just told students to send me an email. Any time they had an issue with a question that they thought may have been marked incorrectly. And what I do is I can find that student immediately on my web page, look at the remark, remark it according to the rubric that I gave my TAs, and then either change the student's mark and then respond to their email and say, problem fixed, sorry about the mistake, or I can explain to them in a little more detail why they got the mark that they did. From a student point of view, you want your mark back fast. That's very important. And when I have a big stack of tests, or I did last year, had a big stack of tests that had to get passed around physically between eight TAs. I had to finish with that stack and then pass it to the next TA. It would take sometimes two weeks to get a mark back to the students. With crowd mark, all my TAs were marking simultaneously. You can see these marks filling up on that marking grid, I guess. I could see it from my desk what was happening. The TAs were getting it done. I said, let's have a deadline. Let's try to get this done by Sunday. This was the test on Thursday. So Friday was spent scanning it in, April scanned it in. Then Saturday and Sunday, my TAs, wherever they were, one of them was in none of it doing some research, but he logged on, got his marking done. Everything was done by Sunday at around 6 p.m. And by Monday morning, the students had not only had gotten it marked, but they were able to log on and actually look at where they lost marks. And they could still kind of remember. Because if it's Monday, you could still kind of remember doing the test on Thursday. But if you did a test two weeks ago, it may as well have been infinity ago. You can't really remember what you were doing. You no longer care is the main thing. And so this has actually turned into a teaching moment for me because they can remember what they did, why they lost marks, and that's something that they're never going to forget. Oh, okay, I got this wrong. I had my physics mixed up and lost some marks, but they're not going to forget that in the future. Whereas if it's two weeks turnaround, there's just not that opportunity. My TAs were estimating it was taking them about half the time to go through and mark. So we scanned 350 students, four pages. That's 1,400 pages. It took her 50 minutes on the clock to scan it. That just got the images on her computer, and then it was about another hour for her to upload all of those to CrowdMark. That scanning time was worth it for sure in that now we have an electronic record of what the students did the time they did it, which can't be modified. It's somewhere up on the cloud, and the students can see it. I can see it, the marker can see it. We can all see it simultaneously if we want to have a phone conversation about it. It's all there on the web, as opposed to being in a physical piece of paper that has to be moved around. So it's a way of archiving their work as well as marking, I think. Yeah, so one question I sometimes get from students for long-answer questions is, are we allowed to write in pencil? And when I first got asked that question, I said, well, why wouldn't you be? Of course you can write in pencil. What if you make a mistake? It's going to get very messy if you keep crossing things out. And students told me that with some teachers they were told if you write in pencil you can't ask for a remark because there's a suspicion there that you've made some changes after you've got it back. And then, oh, I want it remarked. Look, I did it right. And then you get an extra mark. Well, obviously with crowd mark, that's not possible. I've scanned what they've done in pencil and the pencil scans seem to come out fine. And that's what's marked. And in fact, they never even, I don't even give them back their original version. It just stays in a box in April's office. And if they want to view it, they can. What they get is just the scanned version. So it takes away any temptation for any of that kind of shenanigans. Probably about half of the requests that I get for remarks are legitimate concerns. And maybe half of those actually get their mark raised. I mean, mistakes do happen. Especially when my 1TA is marking 350 tests. Sometimes a mistake happens actually either way but you very rarely hear complaints when students get a mistake in their favor. And then the other half of requests for remarks are kind of phishing for more marks that just usually result in no change. So as a professor, I've trusted that my TAs are off there doing a great job marking. And usually the first time I actually look at the marking they've done has been when I'm remarking students' tests after I've distributed them back and I've tried my best to review things, but it's a little hard, especially when you're under a time pressure. And I've found that with crowd mark, part of what I'm doing when I'm waiting for the TAs to finish marking is I'm just scrolling through and looking at the marking work they've done. And that has helped me... I guess helped me advise the TAs on how to be more consistent, especially if I have more than one TAs marking the same question. If I see that one TAs is marking not the way I intended, I can more immediately give some feedback and get them to correct what they're doing. And I actually did that for several of the TAs this time around and said, you know, you gave a six to this student. I really think that should have been a four. You're not really following the marking scheme that I gave and then when the TA had an exact, actual example in front of them, they were able to see what I meant and change their marking scheme. So it just has made, I think, things a little more consistent and fair for the students. I could be looking for something similar, I think, if I couldn't use crowd mark because it's going to be hard to go back to the old way, I think. Well, these are some marked work. So this student got two out of six because they didn't put anything there and there's some comments about what they did, why they got plus two, but why they didn't get these marks. I can go down to the next student on the list who got six out of six just to mark. No real comments when they get 100%. Lost a point here for something. Wrong units, calories per gram, so a little x there. Half a mark there. So I can pretty quickly see what my TAs are doing. That's question two. If I want to move to question three and see how people did a different question marked by a different TA, I can immediately see what each student did, why they lost the marks. So you can either, a TA can actually use the mouse to write on the screen or they can trash that. Or they can double click and actually type comment, Harlow comment. If I was to remark this one, I might put Harlow comment and then type in my thoughts about that particular question, but I can also just cancel that. And then if you want to look at the whole marking grid and see all the green means that it's been marked by a TA. Scrolling down there. And so we can see this one's pretty much marked. Being able to have all of the tests sitting on my screen and sitting on the student's screen, they see their test, and sitting on my markers screen. It's a way of sharing copies of all of these tests so that we are carrying around physical pieces of paper that we have to keep leafing through.