 I'm Deanna Alamon, I'm a professor in the Department of Mechanical and Industrial Engineering here at University of Toronto. So I just started using CrowdMark this semester. My course is MIE 335, Algorithms and Numerical Methods. So there's a lot of math and a lot of computer programming. There are 100 students in the class since it's a core course in our program and I have two TAs to go with the class. Right now the thing that I like best about CrowdMark is just the amount of time that it saves me. Everything from just alphabetizing my exams after everything is graded or after I've collected and before grading, for me with 100 students that could take about half an hour to do. Just turning pages to go from question one of this person, okay, grade it, put it away. Question one of the next person, grade it, put it away. If that takes, say, five seconds of time and I have five questions on the exam and 100 students, we're talking about 45 minutes worth of time just turning pages. So with CrowdMark and the keyboard shortcuts, just click, click, click, click, and I can just quickly skim through everyone's assignments. I like to actually keep a spreadsheet of every single score, every single student scores on every single question. And then I prepare analytics and distributions, histograms, etc., for each of those questions. But entering every single person's scores actually takes me two to two and a half hours depending on the exam. So since in CrowdMark I can just download CSV and I get all of everyone's scores already in spreadsheet format, there's two and a half hours saved right there. So between that two and a half hours, 30 minutes of alphabetizing, 45 minutes of turning pages, I'm looking at close to four hours of time saved without even thinking about any of the other smaller, neat features of CrowdMark. So even if I was only interested in just what's the final exam score for every single person, I don't have to worry about making sure that I added up all the questions right on every single exam because, well, you know, we're only human even if we are professors. And so we do make some mistakes and, you know, this prevents mistakes along those lines from happening and having to deal with students later. And again, I can just hit download on that CSV and just copy and paste that one column into my grade book and then upload it right to Blackboard and I'm done. So the TAs, you know, if you have a class where the TAs are doing grading, obviously there's benefits there. One of the reasons why I don't have my TAs grade in the past is because, well, you know, we have the big stack of exams and in order for the TAs to grade, even if I just, you know, have one or two questions that I feel are very black and white that I feel comfortable letting the TAs grade, in order for them to grade and they have to have the exams in their possession. And so while they have the exams in their possession, that means that I can't do any grading and I like to have very fast turnaround time and while I might push myself to grade, you know, through late hours of the night, I cannot ask the same of my TAs. So by having to give them these exams, you know, it greatly increases the amount of time it takes to get everything graded. Now actually having used CrowdMark, this is something I'm actually considering doing for next semester, is to have my TAs grade some of these questions because they can grade in parallel with me grading and hopefully things can be done in general and faster. From the student's perspective, I did ask all of my students to send me their remarks on CrowdMark after using it and 85% of the class actually responded to me and they said that they absolutely loved it because from their perspective, not only are they able to get their exams and results faster, as opposed to me saying everyone's exams are graded, scores are posted on Blackboard, you know, you'll get your exams in class next week or during my office hours or whenever, they can actually get their exams right then, right there, actually read through the comments and a lot of students told me that before they were not always terribly inclined to really read through the comments, especially if they did badly on a question, it just sort of hurt. But now they can actually sit through in the privacy of their own home, sitting on their own couch or wherever they are and actually read through the comments and the questions that I've gotten back from students to explain, you know, what solution should have been or why particular students' answers were wrong are much different this year than they had been in previous years. Previously it would just be the top students in the class who really want to know everything about the grading and how things went or maybe the students who are really just sort of very social and very comfortable just coming to office hours and talking and asking questions. But this semester I'm finding I'm catching a lot of students that are in sort of like that bottom, you know, echelon of the class or like this sort of middle bottom who are actually really starting to show me that they care about how they did and like they want to understand their performance better and I think that's huge and something that wouldn't have existed without CrowdMark. The way CrowdMark transmits information to students does make them much more engaged in their own learning which of course is going to make them learn better but also I think it helps bring them to a place where they're comfortable with the idea of lifelong learning so that learning doesn't have to begin and end here at U of T. It's not a mysterious process. It's not some sort of black box where they write down answers and like one number pops out at the end but there's a lot of information to be gained about themselves and about their current status and what learning and studying techniques are working well for them and what isn't working well for them, what topics are they good at and what topics do they struggle at which you know might correspond to one question on exam versus another and I think it puts them in a really good position to be basically in charge of their own learning for the rest of their lives. Accreditation is something that all engineering schools have to go through. So by our unit being accredited it means that our students when they graduate are eligible to apply for professional engineering licenses which you know oftentimes are requirements for their careers but the accreditation process is very data and time intensive. Specifically from the instructor's perspective we have a set of graduate attributes that are supposed to be tied to each course so these are attributes that we expect all of our students to have a mastery of at the end of their time here and every course is responsible for you know teaching some of these attributes and part of the accreditation process is to assess how well of our students you know done in these attributes so what we as instructors have to do is tie every graded assignment or exam to one or more of these attributes and then take everyone's scores and sort of normalize them and try to say that okay you know our students are doing well in each of these attributes. Of course the challenge is that every instructor goes about this process in a bit of a different way and it also may not make total sense to say one whole midterm which may consist of four or five questions on varying topics corresponds to just say one attribute or two attributes or three attributes. It would be maybe a lot easier if I could say this question on this midterm corresponds to the particular attribute or maybe even just this part of this question corresponds to something. Now CrowdMark has the capability of adding in graduate attribute assessments along with rubric grading that would allow me to say well if a student did this thing right not only do they get say ten points on this problem but they also get maybe three out of three points on this graduate assessment or five out of five points or whatever breakdown we might want and that way we can in a much more granular process keep track of every student's attributes and then not only will we have better information but it will be more consistently displayed from course to course to course to course and instructor to instructor but if we keep that data over time then we can start to look at historical trends. How are we doing historically? Are we better this year than last year? How about five years ago or ten years ago? Of course obviously projecting into the future but there's a lot of information that we as a university can gain about our students and our teaching effectiveness that can help us in the accreditation process. So the best thing about the actual scanning process was how much I learned about our department scanner and copier. That last iteration where everything worked fine really only took about 20 minutes which is less time than I would spend alphabetizing the exam. So we're still looking at definitely huge gain in time savings of doing the scanner. One thing that CrowdMark allowed me to improve on is my consistency in grading. So normally I have a rubric. Every time somebody makes a particular mistake I write it down. How many points were deducted. Every time they do a specific you know thing correctly I make a note of the comment that I write as well as how many points I attributed to that particular correct component. But you know as I'm going through oftentimes you know one person will will do something I think that is the craziest answer or approach I've ever seen. I don't even know what's going on here. So I'll just take off some number of points and okay just go to the next one but then 10 or 15 students later someone else has made that same mistake. So now I can no longer remember exactly how many points I took off for that previous person. All I just remember is that oh it happened. So now I have to go through all the previous exams. Find that one person's exam. Now I'm now I need to write it down and make a comment because if two people have made this mistake probably there will be more. So I have to write it down go back to the one that I'm currently grading give them a score and continue on. But with crowd mark now I can just instead of actually handwriting all of these rubric comments I can just have them in a spreadsheet and I just type down every single one and I can just copy paste copy paste copy paste every you know plus this many points minus this many points along with the comments and then anyone anytime someone does something really crazy it's now no longer a burden to me to have to keep track of what they just did. I just copy and paste it into my spreadsheet and it's just there. So that allows me to be really fast and really consistent about the grading. So in addition to keeping just a rubric spreadsheet with all the stuff I'd say a great tip for using crowd mark is to learn the shortcut keys to navigate through exams. So I just you know type the number paste in my comment hit enter next you know type that hit you know press number enter and then use shortcut key to go to the next exam same question. So it allows me to just quick quick quick move through all of my exams. Definitely writing a lot is a problem during the exam. My my hand gets tired especially if I'm having to write a lot of comments and I sort of feel disincentivized to leave really long detail comments because it takes time to write them and if I leave them for one student you know if I'm tired by the time I get to the end of the stack I feel really bad if I haven't left the same quality of comments for the bottom students so I end up just sort of you know maybe not just giving a couple brief hints here and there but not really anything thorough but I'm a much faster typer than I am handwriting like basically everyone in the world these days. So now I can really leave very detailed comments and and it takes less time and if I were to actually write things out my hands don't hurt from being out of practice from handwriting and and again I can just copy and paste things to and from my spreadsheet so that everybody from the very first person to the very last person gets equal quality of feedback. I would feel really sad if I was no longer allowed to use crowd mark it just it just saves me so much time you know like all the little stupid parts of grading that have nothing to do with the actual grading. Most of those are just completely eliminated with crowd mark like the organizing the keeping track of statistics making sure I didn't add things up incorrectly making sure that I have reasonable quality feedback consistently through everyone like all of that is basically just implicitly handled and it makes my job easier faster and better quality and you know it's more convenient to because I could just be at home you know watching TV oh you know what maybe I'm done watching TV I'm just going to grade you know or I could just you know pick up and go to the coffee shop if I want to do my grading there and I can do that and it just makes things so much nicer for me and the students have also told me that they really appreciate being able to get their feedback anywhere and everywhere and it would just be I think a travesty if we were no longer allowed to continue using this tool.