 Rwy'n nhw, mae mae'n ychwanegwch angen i gyddiol. Dwi'n ôl yn cael ei gyddiol. Roedd ychwanegwch angen i gyddiol wedi gyddiol. Rwy'n clwrs. Rydyn ni'n brif, ac mae weithio'r gyddiol. Fy o dd �if yn dweud eich gwynedd wrth dyma'r cy sinfularyll. Mae'n ddechrau'r gyddiol. Felly mae'n tynnu'n gwerthwyl sydd am hwyl. from the University of Edinburgh. Hello everybody, that's my first Moodle Mood. Thank you. I'm going to talk about a project that I'm involved at the School of Mathematics at the University of Edinburgh. We are creating our own online assessments for a range of mathematics courses using stack and a system for mathematics in Moodle Quiz. I work with Professor Chris Sangwyn, who is a developer of stack, and Dr George Kinair, who is a lecturer in technology and mathematics education. But they are not here today. So what is stack? Stack stands for system for teaching and assessment using computer algebra kernel. So it's an assessment system for mathematics, sciences and related disciplines. It's an optional plug-in for Moodle quizzes. So I won't feel insulted if you've never heard of it before. Students type and answer us a form of algebraic expression, and with stack we move assessments well beyond multiple choice questions. So at that point, let's think a bit about multiple choice questions. Okay, they are easy to make, very easy to mark, probably ideal for online assessments, but we can't have a whole assignment based on multiple choice questions, especially in mathematics and sciences. So what is wrong with multiple choice questions? So it's a bit like gubbling, isn't it? Like even if you have no idea what is the correct answer, you have a good chance just to pick one by chance, and maybe be lucky and be the correct answer. So in this particular question, I know, yeah, it's tricky, I have no idea what's the correct answer. So if everybody knows, you can tell me. So if I had to answer it, probably I would be by gubbling. So a second reason, probably that's more related with mathematics, is that we can have multiple choice questions that probably assess what we would like to assess. Like here's a very easy example of how to solve a quadratic equation. So the purpose of the question is to figure out that students really know how to solve a quadratic. But is it necessary to solve the quadratic in order to figure out which is the correct answer? Do you really need to solve it? I think not. If you just check the answers, you can figure out which is the correct answer without even solving it, without even knowing the formula. So actually, students, they don't really solve multiple choice questions, they just check the answers and they can easily answer them. So for this quadratic, if you haven't figured it yet, that's the correct answer, but I'm sure everybody figured it. So with computer-edit assessments in mathematics, we want to establish the mathematical properties of an algebraic expression. First of all, we want to figure out that the student's answer is algebraically equivalent with the correct answer and that is also given in the correct form. So x plus x is equal, is algebraic equal to 2x. But would you like to accept x plus x as a student's answer and not 2x? So the computer-edit assessments, they need to be clear enough and distinguish between these two things. So STUG uses the computer algebra system maxima. It can generate random structure questions. The answers contain mathematical content. It establishes mathematical properties of those answers. We can use it for formative and subative assessments and stores all data for later analysis. So I'm going to talk through this point just by using examples of STUG questions. So I hope that I won't scare you and you won't run out of the room till the end of this talk because I believe I picked these ones but I may be wrong. So that's how a STUG question looks like. So by randomisation we may have a random version of this question. So randomisation means we may have random numbers but the difficulty of the question needs to be the same. So in a case like that we may decide the number three to have a different value for each student but the general question should have the same amount of difficulty. With STUG we can have plots which are consistent with the randomisation so they're not static plots. For each random version you can have the individual plots that corresponds to this version and you can have plots at the question text or at the feedback session or at the work solution. A very important figure is that we have line by line reasoning. That's a new feature in STUG 3.6 and it works only in limited cases but we can actually assess the work of the student line by line. We can actually check if the reasoning of the student is the correct one. Not only the final answer but actually the whole working. So let's try to see how a student put an answer. We have a relatively simple integral. So you're a student, you evaluate it and you put your answer so if it's a formative ascension we may allow students to check the answer and see if it is correct or not and they may allow to resubmit the answer. So for this integral has anybody figured the correct answer? You can shout it out? No, maybe not. So correct answer, on over 32. The student puts the correct answer takes feedback correct answer well done. So that's good. So what happens if the student doesn't put the correct answer? Let's say I put an answer like that. Just incorrect answer. Okay, that's not so helpful for the student because the student maybe doesn't know what's going wrong. So actually the computer algebra helps a lot to that because it's better if you have a feedback like that. Your answer is the negative of what is expected. Have you got the limits in the wrong way around? So the student has some hints of what went wrong and can actually resubmit the answer. So in mathematics there are very common mistakes that students do. So we can actually implement this in our solutions and the system can give useful feedback. So another interesting thing is that we separate correctness from validity. So what, me, as a teacher, I may not want to allow students to type floating point numbers. Why to type floating point numbers? They don't really exist, do they? So if a student puts an answer like that, it won't get penalised because it will receive a comment that this answer isn't valid. So we want to penalise the students for validity but just for correctness. So that's a different example of a stuck question with feedback. So you want to factorise a form. So the computer algebra system can actually check the student's answer and check if it is fully factorised and gives specific feedback of where is needed to be done a bit of extra work. The student can resubmit and the teacher can decide what kind of penalty the student will have or if the student will have any penalty or not. So each question comes with fully worked solution so the teacher is allowed to have a fully worked solution and the fully worked solution can follow randomisation. So that's the fully worked solution for the previous integral. So the teacher has access to the live quizzes at every time and can have a look of all the results of every student so it's how it looks like. So you can go and check all the answers that students put. Not only the final answer but all the answers that the student may put and then resubmit it something. So all these data are quite useful in order to analyse them later. We use stack to run an online mock exam and this is a marks distribution. So now we're in the process of trying to see how the marks in the mock exam correlate with the marks in the real exam and we'll try to figure out if it is possible in the future to use stack for exams as well. So this year we use stack for a range of mathematical courses so we use it for like five courses in general, two in semester one, introduction to linear algebra that has 600 students, mathematics for natural sciences which has 150 students, calculus and its applications with 500 students, engineering mathematics, 300 students in mathematics for natural sciences, 150 students. All these courses had weekly online assessments plus proper homework handwritten from the students as well but we're focusing on the online assessments so its course has various ways of assessing students online. For introduction to linear algebra they had two pre lectures reading quizzes every week plus one skill quiz. Some of these quizzes had time, some of them had unlimited time and some of the students are allowed to have one attempt or multiple attempts depending on what its teacher wants to do. So for engineering mathematics we had three quizzes which didn't count for the final mark and one test that counted towards the final mark every single week. And for calculus and its application we had one reading quiz and one skill quiz every week so these were quite a lot of questions and we have quite a lot of data. So for the first run most of the questions were simply like correct answer, well done, wrong answer which is not the correct thing so by analysing all students data between now and next semester we'll implement all common mistakes and then students will have more useful feedback which is really important for mathematics in general. So I think that that was my talk and you may want to prove the following these are algebraic equivalents but which is the best answer which is the one that you would accept as correct answer to your teaching. So thank you. Thank you very much. Questions please. I was just wondering whether the stack actually works with the equation editor in the Atto text editor or do the students just have to type it themselves? The students type, the students type their answer actually, yes they need to type their answer. So they don't use an editor that helps them write the equation? Not yet, not yet. We are open to suggestions. Just following up on that do the students therefore need to learn something about what format is acceptable and also with that do they get live feedback of how what they've typed in has been interpreted? Yes, they immediately see how the answer is deprotated and we have also some quizzes initially before actually taking the quizzes and the courses of how they need to put the correct answers but actually the students actually let's say don't take these quizzes they just try to put their answer but most of the time they have no problem because they can see immediately how their answer is. Thank you. I was very excited by this talk because I'm a software developer who's done and I've helped contribute to stack so I'm quite biased in its favour. It's a very cool thing and I'm very proud to have helped build it. On the subject of different input types the way it's structured is there are different types of input and we don't currently have one that works with a graphic equation editor but that could easily be done. It turns out that's not something that's really needed because actually if you're in the area of maths wherever you input maths into a computer you have to use some sort of syntax so learning to communicate maths into the computer is actually something you need to learn and as you type it's playing it back in sort of typeset form so it's actually a really good way to learn that so that's I think why we don't have an equation editor input it's actually not desirable most of the time that it could be done. Hi I'm just wondering did you look at any other editors Maple TA whereas why did you go with stack? Actually you used Maple TA till recently and we decided to move through stack because stack is more robust and probably it's we want to students to type mathematics we want to take this knowledge somehow and be confident in typing mathematics so we decided to move around online assessments using stack. And would you consider it better for graphics? We are looking at our text editors for maths and different things at the moment and we are considering moving we're using mirrors considering moving to Maple and is this a better option definitely do you think? I don't really know. Neil, hold on, I'll come. Sorry me again. When I looked into this briefly and it seemed like you had to set up a whole separate server to do the stack stuff is that what you had to do at your place or did you find some kind of hosted solution that would do that side for you? Did that make sense? Not really so I'm not in the development behind you know that I'm just... Oh Tim knows. I'll just pass the mic over. Sorry with the geek hat on. Yes it is a pain to install because you need to be able to do the maths this maximum software we use it's an open source computer algebra system for handling the maths. Yes you need to install that and that's a hassle to get it working. Once it's working it does seem to be remarkably reliable our students answer about I'm from the Open University our students answer about 30,000 stack questions a week jumping up to about twice that in any week where there's an assessment deadline. Sorry Tim, follow on question for that is it as much hassle for every upgrade? No. When we were started we were quite nervous about how much hassle it would be to keep it working because it's doing some quite complicated stuff on the server and it's been remarkable how little hassle it's been I don't want to jinx it but most of the time don't you have to think about it at all just sit there doing it stuff. Yep. I'm quite interested in the line by line marking how does that work? How does the middle grade the workings of a student? So it's a computer algebra system behind stack that does this so it goes from one line so you have a correct expression in one line and then you need to have a correct expression on the other line it's quite a new thing we use it only in limited cases like with one variable till now and it's something we really want to develop with any further because that's quite important in mathematics. Just a follow on question to that if the student skips a line and goes down to the next line will the computer program be smart enough to know that they've still got that correct? It depends if it is important or not. Okay, thank you. One more from Tim. I have plenty of time actually so keep them coming. Just to say that I just tweeted a link both to the plugin download page and there is a public demo site online if you want to actually try it for yourself so just find the tweet I just made or come and ask me and I'll give you the link to that. Or just Google it, stack plugin for Moodle. We'll retweet it on Moodle UK Twitter as well. Hi, one of the issues we have with stack is quite a high learning curve for staff to actually develop the questions. How do you do staff development in terms of teaching them how to write their questions in stack? Me, it's me. I have a full time job to write all the stack questions. So you actually write all the questions for your staff? Yes, I do write all the questions for all these courses. I don't know. If you can please do. Hi, it's almost the same question but for students, do you find that students need a lot of coaching to actually take these questions especially with the line by line one to get their answers in the right format? I know we talked about whether they're learning to type maths into the system which is kind of covered but I was thinking about the line by line thing. Do you find it... They have a quiz before actually starting their normal quizzes which is about how to edit stuff in stack but it's quite straightforward actually because whenever they start typing they have an extra window appearing. Probably it was somewhere so... in the previous slides. So you can see your last answer was interpreted as follows so you are quite sure if you are missing a parenthesis or something so you can actually check. So the line by line is quite new. We actually used only two or three questions this semester so they need to actually to actually have a few things like the final form is a factorised and they need to have the two solutions and also have like or or and so if you have here add instead of for it's not correct because if you have a quadratic it's like one solution or the other so for that they may need a bit of more guidance of what they need to type. Hi, thank you so much for a really interesting introduction to your stack questions. I'm interested about the overall grading and the functions that the stack questions would have in your course. So which one? Just what are some of the pedagogical functions I guess for stack questions are you using them for the end the final grade or informative? I'm guessing both but I'm interested in where the grade for the student comes from and what's the role of stack questions. It is used for both so it depends like most of these assessments they count towards the final mark so students have like in its mathematics courses they have 80% from the exam and 20% is from the assessments so it's somehow split between 10% online assessments and 10% the handwritten assessments. So these assessments count towards the final mark maybe not all of them like the seven best of them but they count and it's good to give them feedback so if the teacher wants to allow resubmission like they may have multiple attempts it's good to give feedback so the students can actually have a good mark and also the feedback to be meaningful so that they know what they've done wrong not just put the correct answer or try to somehow guess the correct answer or try to figure out knowing how. Great thank you so they're used for extra credit as well. I think we also in our university had really good rates for students completing optional stack questions so I think it's a pretty great tool, thanks. Last last orders please. Right thank you very much that was generated lots of questions lots of interest, thank you very much. So we have two more presentations