 So, hello everyone, and thank you for being here today in the second webinar of the Moodle Academy. The webinar is about assessment, exploring assignments, and I am Anna Krasa, Education Advisor here in Moodle HQ, and with me I have Mary Kuch, Education Manager here again from Moodle HQ. And together we are going to discuss and see how assessment can be done through using assignments. This session is a combination of a presentation and a live demo. So, you will have the opportunity to see things both from the student perspective and the teacher perspective. And if you want to, you can also log in into the course, the activities will be open now. And access the assignments, the exams that we have already prepared yourself as students. Now, if you want to keep your eyes on the screen and on this Zoom window, that's fine because afterwards you have also the option to replicate those examples exactly as we created in the course. Since we have inside the course, you will find PDF with all the settings of the assignments. So, before starting with the assignment, let's see a little bit a few things about assessment in Moodle in general. So, Moodle can support different assessment methods and different grading methods and has several activity types. The assessment methods can support is actually the auto-assessment, so system assessed by its own automatically. And we have also trainers or teachers assessment. And also we have options like activities for peer assessment where learners assess each other. Moodle also supports grading method, two different grading methods. We have a simple direct grading where grading can be done with a simple numerical scale 06, 010, 0100, whatever you want. Or a custom scale. A custom scale could be something verbal like a missing, weak, fair, strong, or a five-star scale. Or you can choose to set to no grade at all. And also in the advanced grading, we have two options, the rubric and the marking guide. But we are going to see this together later, so I won't say anything about this. As I said before, there are several graded activities in Moodle. We have the human graded activities, as I call them. And they are the activities that require another person, either it's a teacher or a student to assess. And these are the assignment, the forum, the glossary, the database, and the workshop. And we have the system graded, those that are automatically graded, which are the quiz, the lesson, the H5P, and score. So today we're going to focus only on the assignment. And as you can assume, this means that we're going to have more assessment sessions and webinars in the academy. Let's see about how, what are the aspects we need to be aware regarding assessing with assessments. So assignments can be used both for formative and summative assessment. In the formative assessment, the goal is learning. In the summative assessment, the goal is assessment, evaluation. So in the first case of formative assessment, we create assignments that are usually always available and they don't necessarily need to be graded. On the other hand, when we have summative assessment, we tend to set a start and then cut off dates, keep things strict. And also we definitely grade them. In formative assignments, we offer usually additional things and sometimes until someone passes, if we say the pass score. While in summative, we rarely open, reopen a new assignment, perhaps manually. In formative assignments, we don't need the pass grade. We don't, and the completion of the assignment could be said to view or submit. While in the summative assignments, we definitely need the pass grade and the completion criteria is usually a grade, requires a grade. And when we set up this within a course, we often use a 0% way for a formative assignment, while when we have a summative assignment, we use more than 20% usually. It's not strict, but often it takes a big weight. So assignments basically allow students to submit their work for teachers to assess it. This is the very basic concept. And it's one of the most easy activities to set up in Moodle. Yet, it has several settings. And depending the combination of the settings, you can actually create completely different experience for your learners. So together, we are going to see a number of settings and different setups. Before that, let's start with a poll that help us know each other. Let me initiate this. Do you see it? Yes, I can see it, Anna. Thank you. So let's all have a go at the poll. There are three questions. Tell us a bit for your background, what are you doing? And what are you using Moodle for? How long actually? And if you have ever used assignments. It's very interesting to see the answers here, Anna, shortly, yes. And as a reminder, do feel free to type your personal experiences in the chat and questions as we go through the different types of assignments as well. Thank you, Mary. So I think I'm going to end the poll here. Okay, 15 people, 16. Oh, a little bit more. Let's wait for the last one to participate or perhaps not some. Okay, I'm ending the poll and let me serve the results. Do you see them? I can see them, yes. So mostly trainers we have here and we have mostly people more than two years who are using assignments often and regularly. That's good because that means hopefully we should have quite a few comments in the chat, Anna, as we're going along. Yeah, it seems that we have a very experienced trainers here, most of them, and I hope that you will find something interesting in this webinar. So I'm stop saying this poll and let's move on. And we will begin with the assignment submissions. First of all, we can create an assignment that requires no submission. This assignment could be used to assess an offline event. So in this case, learners are here, don't need to submit anything, but teachers can see a list of the names and offer grade and feedback. We're going to see an example with no submission required, and we're going to have an activity with a date set, no submission required and feedback comments, enabled for the teachers and the grading will be done with zero 10 grade. Let's see it. If you want to, you can access the course and see the section one, no submission required. And our first assignment is the archaeological museum visit. We assume that we are in a school in a secondary school and there is a trip educational trip organized by the teacher visit to the archaeological museum. The task here for the learners is to participate in the event that will be held in the museum. So as a learner, there's not much you can do all you can see is the dates and perhaps leave a comment if you want to, but there is no actual submission. So let's take a look. As a teacher now as a teacher, I can see I have two options that you all submission and the grade. Let's take the grade. Now, here, I would have a submission but I have the note that doesn't require a submission. So I can simply give a score and save grade work. And I can simply save and so next or just save changes. And here I can notify I can select to notify or not learners. If I antique this and save changes then learners won't take a receipt won't take an email notification saying that I have reviewed their work. So let's go back. I would like to hear from you. Would you use a no submission assignment or have you used before actually have you used one Mary. Yes, so while people are typing in the chat hopefully you can give us your experience whether you've used this type or not. I can say that I did use to use this quite often when I taught languages in a high school, because for speaking practice this was obviously in a blended environment we had classes. In the school and then we would set homeworks and so on online with Moodle. So I would assess their speaking face to face in class, but then I would use the no submission assignment to grade them. This was actually before the time when you can now record your voice audio or video through ATO. So this was very useful for us too. Also as a kind of a attendance sheet if you're going on a on a trip just to check that students are present. So do we have anyone who's actually tried this and if not it will be interesting to know if we might have persuaded you to give this a go so we'll just have a quick look in the chat. Kim is saying I've used it she's used it for offline graded activities as you have presented. Yeah there are two ways you could grade offline activities, you could go straight into the grade book, but actually this I think is a much simpler way of doing it and is more accessible for your students to know the grade that you've given them. Kim is saying to give some points for class discussion participation or attendance to be to appear in the grade book. Gemma says she used it for attendance, but also for preparing to read a file that would be discussed in class. So that's some ideas there for you Anna. Thank you all so much. And let's move on and see another type of submission for our assessment assignments. Online submission, usually with most of the in most of the cases we usually ask learners to make a submission and turning on the online submission we actually asked learners to write something inside moodle inside the course inside the assignment for them to review and grade. And when I say right doesn't mean literally right anymore. Since the field where users add their text contains the other editor and with the other that we can now have and rich text with all the stylist will want and links and icons and images. We also can directly record audio and video up to two minutes so we can have multimedia submission actually. But everything done through the other editor. And we are going to see a second example with online text we have enabled a world limit of 100 words and in this assignment we require learners to click the submit button. It means that learners can can make submissions and their submissions will be kept as draft until they actually click the submit button for teachers the comment in line and the feedback comments is are enabled and also PDF annotation. We have set the grading up to a point scale symbol direct grading zero 10 scale. So let's see it inside the course you will see, you can see the second section online text submission, and now our assignments reflection. You can see that the description, of course, and the add submission. So, when a learner makes a submission. Then the submission status changes, and is shown as a draft not submitted, and it won't be submitted until learner clicks actually the submit assignment. You can, this means that your learners can work on their submission for for long time, you can begin this activity, this assignment at the beginning of the semester and close it at the end of the year. So they can keep coming again and again and again editing editing, and when the field they are ready, they will click submit. So, normally I would show you the PDF annotation in my in the life set in the life side, but I've noticed that it's a bit is slowed down my machine because I'm sharing my screen so I'm going to show you a couple of screenshots. As we said, in this assignment we have enabled both PDF annotation that's why we see all these left window, and we also have enabled the inline comments. So let's take things one by one. Let's focus first on the PDF submission. If we just want to work with the PDF submission, we can simply click here, the little icon and change the layout of our screen and make things more clear. When we use a PDF annotation, we can leave comments and create different colors of comments. These are items are clickable and when you click them you see the full text in and you can draw or you can make underlines and other shapes. And also you can highlight, like here in different colors and use stamp as well if there are some icons that you can use a stamp. Now, let us see this inside the course. Let's go great something. As you can see, the PDF window appears, but of course here we don't have any submission. So let me collapse the PDF annotation panel and focus now on the submission. Let's find an activity. Let's find an assignment actually that needs to be grading and I'm filled in here if it requires grading. I don't have my friend learner one has made the submission I can read here. And I can give the score. I was thinking. And if I want to, because I have enabled the inline comments, I have the ability to see the submission text inside in my feedback comments. This means that I can come here. And perhaps in italics, leave a comment. And I can just notify my learner saying comments in italics and I can save and move on to the next one or I can just save changes and that's all done. So now let's go back to a presentation. And I want to ask you PDF annotation or inline comments, what would you prefer? What you would prefer Mary and what you all prefer. Okay, well, I can tell you what my preferences in my experience and then we'll see what people are saying in the chat. I'm a big fan of online text because again, with teaching languages at a fairly low level to sort of 15 16 year olds. They never had a lot to type only a couple of hundred words. So online text was perfect for them. I prefer the inline comments because I like being able to go into the feedback box and cross things out or using the editor or highlighting things in a different color. Although of course we have to be aware of accessibility and you know the concerns about explaining mistakes by using different colors only. But let's see what people are saying if the if other people prefer PDF annotation. Okay, so Gemma prefer prefers PDF annotation it'd be interesting if you could tell us why Gemma Kim says both depending on what she's assessing. And of the can makes a very good point about preferring PDF annotation especially when visuals are including. And Kim says using PDF annotation to grade or to mark up imaging submissions, whereas rather than agrees with me prefers online commenting is easier and quicker. Some people sometimes have issues getting the PDF annotation to work but most times it can be very good if, especially if you're a fan of stamps or the comments and so on that Anna demonstrated in your screenshots there. I think that's it for now Anna. Thank you. Thank you so much. Let's move on. The other type and the type of submission we have is a file submission. In this case learner submit a file for teachers to assess. We are going to see an assignment where we only allow document file. See teacher has the right to the capability to restrict the specific number or the specific type of submission allowed. So here we allow only ODP and doc files and learners have to accept the submission statement. This is interesting because this is a warning for learners that a reminder that be sure that this assignment that you are this work that you are submitting is your own sense not borrowed from somebody else. Here we also have set up additional attempts to open automatically until pass. We have a simple direct grading with zero 10 point scale and pass course set to a so feedback comments and files are enabled here and offline grading works in and figure files are also allowed. Let's see it. You can access the section three file submission. And see the article review. Here we are in a university and we ask learners to submit an article review about quality in education and they have to write a thousand five hundred words following the app of style for those who are familiar with the academic writing. And learners all they have to do is to click and add the submission. Please note that the attempt number we have this attempt is number one. And this is interesting. Let's go see as a teacher. And great one activity. So let's click great. If you remember I had already filtered to see what requires grading here. So I'm seeing the work of learning one and I see the file here I have to download the file, read it and give great make my comments perhaps send back feedback with comments file with comments and at the bottom it's interesting to see that you can see the attempt settings and see which number of them is this and how many attempts are still left. And that's kind of things. But when I have personal that's my this is my experience actually what I like to do when I have submission fights, fights as submissions, I prefer not to go through this page and grade one by one, but I prefer to view all the submissions and work from here. Now, here you can you have several options very interesting options. For example, if you have connectivity issues you may download all the submissions and review them offline. If you can download the grading works it. It's a CSV file at downloads I can show you how it looks. So you get a CSV file with users name and mail address and submissions. You can write the grade and the feedback comments. And then from here you can upload it and release all the grades and feedback at once to everyone. And similarly you can upload multiple feedback files if you want to. Now, from this page I can filter to see what requires grading or what is a draft or check several options. I'll leave it to grading and I have the ability also for a quick grading. When I click the quick grading, you see the grade here is opens and editable field as well as the feedback comments. But this time I think my friend later on one didn't do very well. Please review your work and resubmit. And when I'm done with all the notes I can save quick grading changes. Now, if you remember, we have set up this assignment to automatically reopen until pass grade is reached. That's why my friend who needed to accept the submission statement before submitting. When he got five, he didn't reach a five out 10, he didn't reach the passport. He actually failed. And when he fails, the attempt reopens automatically. It's marked as a second attempt. And he can see that he can add a new attempt based on previous submission or start a new submission from scratch. He has those options. Now, here in this example, we have seen two things. One is the submission statement is the line that they need to click to in order to make the submission and the reopen until pass. And I would like to ask you, would you use this? Which one actually would you use for formative and which one setting would you use for summative assessment? Mary, what do you think? Well, I'll see. Let's see what people say in the chat. I can give you my experience of submission statement and not reopening until pass while people are typing actually. This is exactly what the situation was when I did a master's a couple of years ago, which was on Moodle using Moodle, which was great. And of course, we had to upload for each of our summative assessment at the end of each semester, a word document. So it was restricted to word documents. And we had to agree to the submission statement. And then of course it went through a plagiarism detector as well. So that was just one grade. So in terms of reopening until you pass, what do people think in the chat there? Would that suit formative or summative assessment? And does anyone have any experiences or thoughts on that? Do you always use a submission statement? We use this in Moodle Educator Certification, for example, obviously to ensure that what people submit for their certification is their own work. Kim is saying, right. So Kim is saying, I agree, submission statement for the summative and then reopen until pass for formative assessment, which I think is what you would expect, Anna. Yeah, I think so too. But we can always be creative in the software settings, depending on the context we are working on. Absolutely. Everything depends yet. So let's move on. So far, we have seen cases where one learner was doing an individual submission for teachers to review. But assignments don't have to be an individual's work. Actually it can be a collaborative work. If we create groups in our courses, we can set the group submission on in an assignment. And in this case, we will have the group members collaborating inside the assignment and making a final submission for teacher to grade and provide feedback. And of course, the feedback goes to the whole team member. Now, we have two options here regarding the submissions. We have the option where one of the team members requires to make the submission. And we have also another option where everybody from the team requires to click the submit button. Of course, all the members of the team do submit the same thing, but requiring to click the submit button. It's like he actually asked the learner to confirm that they are okay with the work submitted. So we have set up an example with online text requiring, requires learners to click the submit button. This means that learners can keep drafts of their work until they are ready. And then each member of the group will have to click and make a submission. And we have feedback comments for teachers and a civil rating with zero landscape. Now let's, you can see, but in this case, please be aware that this assignment has been set up to accept submission from people who belong in group. And you are not belong into a group. You don't belong in a group. That's because this course is a self-paced course will remain open as a self-paced course. So groups won't be maintained. We only have one group and we have our two test learners inside so you can access it but you can only actually replicate it in your site with the settings provided. So here we have our friends London one and learning to and they are both members of a group pay. And they have made an online submission and as you see it's a yet as draft so they can still edit it and after the initial submission. Both of them, both of the learners see the online text and see exactly the options to edit the submission remove it or submit the assignment. So, while the assignment is still in draft, they can come and edit again and again one after the other multiple times and work on the same online text submitted here. And when they are done, they will have to actually click the submit button actually they will have both of them to click this submit button. Let's see this from a teacher's perspective. So, we are in a corporate learning environment this time, and we have same department colleagues group together, and they have to do a SWOT analysis and set up a smart action plan, very typical things in a corporate environment. As you can see, the setting requires group to make a submission. So, either you are at the system gives me a warning that some people are not member of a group, or they might be belong in more than one group so there is a problem for the for them to make submission. But this is a deliberate deliberate setup. Let's go great. As you can see I have two users here, and I can see their submission. And I can give them a great normally. Well, actually, they didn't do very much work. And I can give them my feedback for the work. And at the bottom, you can see we have the group submission settings. And we can choose to apply the grades and feedback to the entire group at once. So whatever I write here goes to both of the learner one and two, or give them separate feedback. Now, with this setting and this option in mind, I would like to ask you, would you give the same grade and feedback to all group members, or would you prefer in a group assignment to give different feedback and great to its team member. Mary, what do you think? Well, I've got strong feelings about it actually, but I think the, I think the answer is probably it does depend on your learners and your situation and the task that you've said. But when I use group assignments, we didn't use online texts we used to get, I would have a class of say 30 we would have groups of about five. We would each upload different types of files that together made a whole package. And I had with the teenagers issues with some group members working harder than others. And so I insisted that they all submitted each group members submitted, and then I graded them separately, even though it was actually a group assignment. But I do appreciate that in certain circumstances. And again, as in my masters we, we worked in one of our modules, we worked in a group I think it was a group of four. And that in that one we all got the grade together. So be very interested to see what people in the chat, what your experiences of whether you give the same grade and feedback or different. Or indeed, if you've used group assignments, or if we might have persuaded you to try them. We're just waiting to see if anyone has any comments there. At the moment, I think we don't have any comments on group assignments that's interesting. Perhaps it's not something that's used a lot by the people in our chat. Jessica. So you can give the same grade, but you can allow group members to peer assess. And in fact, this is something that happened in my in my masters as well. So that feedback can increase the individual group members grade. And it still says without a feedback sheet it's hard to know who did what on a group project. Yes, it depends on what the task is that you're setting them for example, if it is an online text assignment, then how do you know who's written what, unlike in a wiki for example yes. But anyway, that's that's that for now and thank you. Well, I have to say that I, I had a really bad experience with group assignments, not, I think it's not a problem with teenagers I think we can find this in any age actually. People don't contribute to say, and it's difficult actually the assessor the cannot pretty tell who did what usually. And in this case is yes as Jessica said, a feedback for peer assessment peer evaluation of the experience it's perhaps the, the best option. Let's move on. And let's see for the assignments the advanced grading options, what we have. We have a marking guide, and when we use a marking guide teachers enter a comment per criteria and the markup up to a maximum. It's a bit complicated but we will see this in detail. The point is that when we use a marking guide and advanced grading method, we actually don't change the flow of the assignment, we just give more options about grade and feedback to the learner to the teacher. So, we are going to see an example that is the requires an online text submission, and a file submission is also enabled. And we are going to assess this with a marking guide, the maximum grade is eight and the past score is set to eight. We have this example at the sections five assignment with marking rubric, you may check it. And I have to tell you when teachers set up the marking guide has the ability to so or not the grading criteria with the learner so if he does, if they do. The grading criteria will show up before when a learner comes in, and I personally prefer this as give some help to learners on what is expected from them and what they need to cover. So you can see in both the criteria and some explanation text and the maximum score they can get. Now let's take a look as a teacher. Let's go grade one. So I have the submission here. And afterwards I have the grade section where I can see the criteria here for pre criteria. And also I can see some comments. In fact I can see comments that the comments that teacher created for the learners, and I can hide them if I don't want them, and also descriptions for teachers. I found this very helpful to be honest, because they give me an idea of how this criteria must be assessed. What should I expect as a grader for learners to write about this. And I guess come here start typing my, my feedback, or I can click the insert frequently used comment and select one and give my grade here for the first criteria, and then check. What should I do even better here. And this is actually very convenient and it really speeds up grading and make it very specific very clear. So I can move on assessing everybody else like this. Now, do you use marking guides in your work. Do they fit in your case. Yeah, I remember training it was I think the history department and explaining about marking guides. They thought it was a good idea but it seemed to be a lot of effort to set it up initially. And they showed them about the frequently used comments and they were completely won over, and they loved it because of course especially if you have many students, and also if you're several teachers working in a department, it allows you to be consistent in how you are grading students between teachers, and also it does save you a lot of time, and you agree beforehand what your frequently used comments are. So it's a great time saver, once you make the effort to set it up. Yeah, so we have. Yeah, people are fans of marking guides says she uses marking guides when she has frequently common feedback. Jessica also always like marking guides and Kim is suggesting agreeing with me that it does take some time to set up. But of course you only need to do this once, and then you're done and can hasn't used it but does like it maybe we can persuade you then to try that can. Excellent. Thank you so much. And another method advanced grading method is the grading rubric. Again, we are focusing on the stage of grading and feedback and this is what changed. The techniques are basically a table like this one where we have a list of criteria and different levels for evaluating its criteria. So, we are going to see an example with online text and world limit setup with rubric of course the maximum grade is set to 10 and the past score is eight. And you can see it in the six, six section assignment with rubric. Again, as a teacher I have chose to serve the grading rubric criteria and levels with the students in advance. So they have in mind what, how they're going to be assessed. Sorry, let's go check it as teachers, this one, and grade activity. I have here the submission. And below that, I have the grade section, which contains the whole rubric. So I have to give a score about originality. Yeah, not very good here. And also I can give feedback for this specific criteria, tell them what to do best and how to improve their work. I can give them for the start something good language was great vocabulary was average. And I can save and so next, of course, if I want to provide an overall feedback for the whole work. I can also name the feedback comments, and I will be seeing at the bottom, an extra feedback section. But please tell me, are you or will you consider to use rubrics. Well, I can tell you Anna that I have used rubrics in my past and present life. In fact, it worked really well for assessing writing as a languages teacher because the examination criteria were that you had to grade students on accuracy. Of course, content, did they actually answer the question that was set in the writing a task and style as well did they use an appropriate style or register. Moving on from when I was a teacher and coming into moodle of course, Anna and I and our MEC moodle educator certification facilitators, we use rubrics to assess in each of the courses bring building up to the certification. And as it has been said here, they were very popular with students, students like it because they get detailed feedback, but it's also very fast for teachers to operate. And it's another thing like marking guides, as long as you take the initial time to set it up, the benefits then are manifold. Yes. And Emma says that they recommend using it, although don't see much evidence of it in use. And Caroline says it's an easy way to clearly demonstrate the assessment expectations to students it is a worthwhile time investment. So I think everyone is in agreement there. Okay. We are all agree that it's really worth the effort. That's, that's for sure. I think it gives clarity and it manage expectations and set the fairness I mean between a learners and assessors. Yes. Moving on to our last example. And this is something different that we are going to see. We're going to study marketing workflow. So far we have seen one learner or multiple learners to make one submission for one teacher to review and get feedback. But now we're going to see learners make a submission for several teachers to review the assignment. So, marking workflow actually creates an internal cycle between teachers regarding grading. So we have an assignment submitted. First teacher reviewed gives a great feedback and pass it over to a second teacher who gives again a gradient feedback and this circle can make several rounds until both more teachers agree. And they are ready to release the grades and feedbacks to the learner. So we are going to see an example that requires online text with the word limit setup. File submissions are enabled and requires only images and feedback comments are set up for teachers commenting line as well great is a simple direct 010 scale. Marking workflow and marking allocation are enabled. Let's see assignment with marking workflow section seven. And for learners what they actually see it's not really impressive. Perhaps the grading status is what is interesting. It's the in outsource no market when things begin but as the marking workflow moves on it can be say that it's in marking assignment is assignment marking is completed or it's in review. It's ready for release or it's released. Be aware that learners will only be able to see the feedback and the grade when released status is set. So let's go and check the becoming a journalist. So you are in an online course about journalism and you have to write to cover an event to be published at your local newspaper. So you're happy will be reviewed by your trainer but also will have to be evaluated by an editor. That's why marking workflow is enabled here. And I can see the event. You can see the images. That's the whole submission. I have a simple direct grading up to 10. I think this is average needs some fixing. And I have here the marking workflow state. I would say this is in marketing because I already gave a grade. And after giving my feedback here. Let's say I think that all things need to do be said or whatever. I can choose another greater actually the reviewer who's married to pass the assignment over. So this this assignment will remain in marketing until married review. We may change our opinion several times and keep this state in marketing. But when we will be ready for release and we will be ready to share our thoughts and feedback with learners who will change it to release. And saving and give it away. So I'm not sure if anyone has tried this. Would you have you tried or would you think of using the marketing workflow. Very heavy you I haven't actually I can see how it is really useful especially as you say. If you are in an organization where you need someone to moderate or or verify your grading to ensure consistency. Claire is saying there that this looks great for moderation I agree. Kim uses it and I've heard of this also she uses it to release all feedback at the same time. So I think we might be getting some fans here who would be interested in trying marketing workflow. And you're right I'm not sure many people have actually used it but it has a lot of potential there. To be honest it was one of those settings that I've been looking for. I've been seeing them in the list of assignment settings I was I wasn't daring to take a look. But when I finally did I was very impressed of the capabilities that gives you. I was asking a question here do all submissions need to be marked by multiple tutors. Or could you select a few to be moderated but the rest marked by just one person I think the answer is you have a lot of control over who marks what and who moderates what don't you Anna. I don't know if you remember when I click to allocate to another marker. I could see my own name as well as yours so I could keep it on to myself so by that I could just say it's me the marker it's allocated to me and I can just choose to release it so you don't have to review it. So it's quite flexible if you use marking allocation, a marking workflow with marking allocation, you can select and decide what to do, and who does what actually. So thank you so much for this. Let's go for our last poll quickly we just are in the hour. It's just a wrap up for. Let me know when you see it. Yes, I can see it now Anna. Perfect. So, what, what we would like to hear from you is, which of these settings that we have seen today are for completely new to you. I don't know if there will be anything new actually because you were all quite experienced. And which is the first thing that you want to try with your learners. Okay, while people are answering can has a question actually does the allocation selection notify the grader that they have items to grade. I'm not sure it does actually. Yes, I think when you, when you look in and you see what needs to be assessed you will see those that are allocated to you, I think that's how it works. Okay, thank you. Well, we see how we're getting on with our poll shortly. Okay, it seems that. Let's just wait a few seconds more. Okay, and I'm releasing. Ending the poll and starting results. So, right, group submission PDF annotation offline grading. Marking guide rubric marking workflow are quite new marking workflow actually is the most new I'm very happy that we saw that today. About the first thing that somebody will try is the group submission PDF annotation quick grading marking guide rubric and marking workflow again, the winner will say marking workflow. That's great. That's to last yes. I'm really happy to know that to hear that. I'm happy actually that you found something new in this webinar. And that was it actually our assessment exploring assignments webinar has now come into an end. Thank you so much for being here assignments are quite diverse have so many settings and we haven't see them. All of them, for example, haven't seen the anonymous admission and we haven't see the high graders identity for from learners. But I think the possibilities are endless and definitely encourage you to experiment. And you can come from the course, download the PDF with the settings and replicate the exact activities that we have here. So thank you so much for being here. Thank you.