 My name is Alfonso Gracias Az. I'm a lecturer in mathematics at the University of Toronto. I have used CrowdMark for a first year course. It starts with over a thousand students and we have five instructors and ten TAs. The course has four tests, every test has about ten pages so you can multiply and that's the reason we use CrowdMark for it. Roughly five sections, two hundred students per section. At the beginning, so the course starts with that, a thousand students. The course ends with a smaller number, about eight hundred students. The most apparent advantage, the main reason to use CrowdMark is that it saves time. It saves time and it also simplifies the marking process and it actually reduces errors in the marking process but all that goes together. I would say roughly that on any given test, if I were to do it as a CrowdMark exam versus a non- CrowdMark exam, I save maybe half the time. In a class with eight hundred students, any process that will save you one minute per student when marking, it's about thirteen hours total so it's not something to sneeze at. So CrowdMark allows us to save time by removing all the parts of marking which are not marking but it also makes the process simpler since not all the markers have to be in the same room at the same time. They don't have to exchange papers, they can mark in parallel the same papers at the same time and it also reduces errors in the marking because in the traditional way we mark a question and we have to copy the number somewhere else and we have to add them then we have to introduce them in a computer and when you're doing this for a thousand students, if it's a human doing it for a thousand students, there is bound to be a few errors. Every course I've ever graded, every mega course like this I've ever graded, we've made errors when tallying numbers and with CrowdMark there has never been an error of course. For the students, one thing they love is that they can get their results, the results of their test immediately via email. The process of returning 800 papers to 800 students traditionally is not a simple one. Imagine alphabetizing this big pile and then returning to a lot of the students coming at different times and papers are always lost and students are always claiming the paper late or whatever. CrowdMark allows us to give the papers back to the students right away and they can see exactly the results immediately and that is a big advantage. It doesn't sound like a lot but if you have run across a traditional way you realize it's a lot. The other feature students like is that it makes their regrading process much simpler. When we've made a simple error in marking, a student can notice it and can request this to be corrected by email because I don't have to be in the same room as a student, we don't have to have the physical paper and that is much simpler to do. So we've saved a lot of time but also we streamline the process of regrading and that's the second thing students like. Using CrowdMark has removed an important source of cheating which is doctoring papers. When you give a paper back to a student and the student comes back the next day saying, you forgot to mark question 5, there's always a question of did they add it afterwards and the fact that we don't have to worry about that also removes, doesn't put us in an adversarial mood with the student which is a good thing. From the point of view of the students receiving the paper quickly, eliminating this source of conflict of cheating and simplifying the process of regrading is a big advantage and I think the students are overall very happy with it. But there is yet one more reason I particularly like CrowdMark which was not apparent to me until I started using it. And that is that CrowdMark allows me to create better tests. When doing a mega course like this one and designing a test there is always the concern of am I going to be able to grade this question and how long is it going to take me. There is a type of question that measures some high level skills that are more creative and open-ended which by themselves are going to be very time consuming to mark but in addition they will benefit from being marketed in a way like with CrowdMark that allows to flag some papers and separate them and mark some of them first and coming back to the others. Without having to do all of this physically with papers. These kinds of questions that measure high level skills and that open-ended are creative are normally always absent from any large course and I think this is bad because that's one of the learning objectives of the course that we should measure. And ever since I've been using CrowdMark all my tests contain questions like those which as I say I had never used those kinds of questions in large courses before only in small courses. So overall CrowdMark allows me to create a better test because I can think of what could be the right questions to ask in this test so that the test is fair and accurately measures the learning objective of the course and I do not have to worry about and can I mark it in time as well. Normally my TAs, my markers and my extractors on myself I guess we are all heavy computer users so we like to use keyboard shortcuts and we like to use all the features in there. I've heard from some of these people at CrowdMark that some users actually don't like to do this but I find that the keyboard shortcuts are great. In particular when I'm marking a test, normally there is I don't know in a question I have five common errors and for every error I would like to write a lengthy comment which I'm not going to do by hand if I'm marking on paper but I can write my five comments and put them in a Word document and then every time I come up with a paper that makes error number two I can copy-paste it and write a lengthy explanation to it. If in addition I'm using the keyboard shortcuts that becomes quite fast. Actually I prefer the keyboard to the mouse but I know some people prefer the mouse. In that sense. Flag in Exans is also a good idea. There are some type of questions that will benefit from doing a run-down very quickly through all the papers that are easy to mark and then come back to the papers that are hard to mark and using the skip mark button on CrowdMark allows you to do this very easily and if there are some that are more complicated it can be easily flagged and then be sent to the coordinator or the instructor or whatever. In my first test I did the scanning myself and I did scanning myself because I wanted to understand exactly what the process was, what was involved in it before I asked the TA to do it. It's not fun. But it's also not fun to alphabetize paper or to enter a thousand numbers on a computer screen. It's one of the many manual tests that come with the job. In the last test we actually outsourced it to a professional company that does that and that certainly is a better process from my end. It costs some money but if one counts how much time we are saving by using CrowdMark and we turn that into budget, into TA hours that we have to pay for marking overall we save much more money than the money it costs us to pay a professional company to do the scanning and it's preferable to give it to a professional company because also we can be certain they're not going to make any errors they're not going to miss any page and they have more capable scanners than the one we have in the department. But the main surprise I discovered using CrowdMark as I mentioned earlier is that it allows me to create better tests and this I was not expecting. I jump into using CrowdMark because I saw it as an opportunity to save time and my TA budget is limited so anything is good in that sense. Any hour a TA doesn't spend marking the TA can spend it helping students in the method center or in a tutorial or anything like that. So I jump into CrowdMark because of the time saved. I did not expect that it would also allow me to create different kinds of exams with questions of other types that I would not dare to include in a paper traditional test. I guess that counts as a surprise. The other surprise is more on the human side which is I did not expect the people in CrowdMark to be so quick and responsive. I'll say I've been giving CrowdMark a hard time ever since I started using it by asking them to implement all kinds of features and to improve things and to change things and they've put up with that very well and they've been implemented all the things I've been asking them very quickly.