 Yeah. Hello everyone. It's the time of year when our minds turn to the evidence we have over student learning. What do final exams mean during a pandemic? Headache. You may rethink your final is under the current unprecedented pandemic situation. And Judy, please click on the next one as well. There's animation and raise a question about whether you use your approach is the best one in the first place. It's a multiple choice exam the best way to measure learning. If not, what other options are available? And then yes, do I have to give a student more flexibility? Then when it comes to grading, it's going to mean a lot of work for them. But would it be worth it? You can click on the next one. It's more. Yeah. And then final one. Yeah. You may have a lot of questions and challenges and concerns about your final exam. Today we'll explore various assessment methods, different strategies that can be implemented online. We are super excited to have you here and excited about the conversation we'll get to have today. So Manuel. So I just want to take a moment to the territory acknowledgement. And so today I'm coming from the Muscovam and city territory. I live on the UBC campus, which for centuries has been a place of learning for the Muscovam people. And I usually share this image with the participants just to explain that when I work on campus and I see the totem poles, it means a lot to me because as you can hear, I'm coming from a different country. I'm from France originality. So it allows me to really understand all the history, all the background of this particular place. I really understand that there are a lot of things happening, a lot of cultural elements that I may not be aware of. So a totem pole like this one, the reconciliation totem pole is actually quite important to me because it is a way for me to fit that gap that I may not know about all the history behind. So I just want to take this moment to kind of always reflect back on where we sit and the importance of always acknowledging where we come from. Next slide. Judy. Oh, introduction. My name is Sunah Jo. I'm a faculty, CTE faculty liaison for the solar school business. Judy, you muted. So my name is Judy Chan. I work as an education consultant at the Center for Teaching, Learning and Technology. I'm also a faculty liaison with the Faculty of Land and Food Systems and I'm also taught in the summer. And hi, everyone. My name is Jason Myers and I'm the faculty liaison, CTE faculty liaison in the Faculty of Arts. Right. And Manuel Diaz, I work at CTE. I'm also a faculty liaison with the Faculty of Science. So we just want to welcome to this session today. So to get started, we usually have a little activity to kind of get to know you a little bit better and see how you're feeling approaching your final exam. So I have this little poll question that should just pop up on your screen right now where we just want to know how you're feeling about your final exam. So if you're ready to go, if you're terrified, excited, or maybe some of you don't even want to think about it until next term. So I'll just give you a few seconds and then we can share the results. So I have two more voting. All right. One more. So we have the perfect distribution. So some of you are ready to go. One of you is terrified. One of you is excited and one of you is actually don't even think about it until next term. So we have a very good distribution of participants today. But feel free to kind of always ask questions. We're going to be also using the chat right now, actually. So you can actually get to know your expectations a little bit better. Great. Yuri hoping to find a way to balance between keeping academic integrity, assessing students learning. All right. So I can see academic integrity is a big concern for you guys today. Andrea securely open book closed books. Aha. Okay. That's a good question. All right. So rest assured, we'll be talking about all these different elements. Maybe not in really, really detailed, but at the end we'll be making sure to talk about the key considerations for you. So certainly talking about security and exam integrity, that's a big thing. Maybe another question for you in the chat. I have three, maybe somebody else might be missing. Well, Grace, you already mentioned. Okay. Yeah, I think we can move on to the next slide. Thank you. So we do have... Okay. Thank you, Andrea, for Proctorio. We'll be talking about Proctorio in a little bit. So I'll be giving you a sense of how it is being used. So basically by the end of the session, you should be able to choose one assessment strategy that can be applied online. So from what I can see, some of you already have an idea, but you're really interested in the integrity of your exam, applying various design strategies to design an assessment that can help maintain exam integrity. So we've already anticipated this and then identify effective techniques to assess student learning online. So we're trying to kind of cover all these things today and really make sure that you live with something that can be applied right away online. So this is just a list of... There's many different options you can use for delivering your online exam. And here's a list of some of the options that we've heard from instructors that they're using. So many of you have indicated that you're doing kind of a traditional timed exam using Canvas quizzes, multiple choice questions, or short answer essay questions. Take-home exams are another option that a lot of, in certain contexts, work well. Or a combination of take-home and synchronous exam is, I think, similar to what Andrew is describing, where there's an open book component and then a closed book component within the same exam. You could also do a similar thing where you have an open book take-home component for a case study or something like that that you want students to have a little bit more time or writing exercise and then a timed synchronous exam. Similarly, the in-tray box exercises, if you're doing something like a case study, some instructors have found that distributing the case study ahead of time for students to look at and review and analyze on their own before coming to the exam is an effective way because when students get the case study right in the exam, sometimes there can be a lot of anxiety and they're rushing to try and understand it. That if they're given it ahead of time, they can take a little bit more thoughtful approach and then the questions that they ask about it. The students won't know those, but they'll have a chance ahead of time to study the actual cases. Oral exams and then we've just learned about recorded oral exams with an invigilator, which is a new one that I haven't heard about before, case studies and two-stage exams. These are just some of the options you can use if you're concerned about whether the multiple-choice timed exam will work for you and assess the student learning the way you would like. There are options for you and so if you have questions, bring those up and we can go through some of these a little bit more detail later. We just wanted to kind of get some ideas out there to get you thinking about what's available to you. Go ahead. All right, so I guess that's me. So in the next little while, we're going to have some interesting discussions on different elements regarding online exams. So to help you with this, we've identified an interesting article written by Laura Kealan, who is a nursing educator from Northern Ontario. And what she's been doing is just showing her experience designing or redesigning her online exam or her exam to make it fit for the online delivery. And she's been giving 10 tips to kind of explain a little bit the process she's been going through to kind of really design something that makes sense for an online environment really. So she's been really considering integrity, which is also a big concern for you guys. And she's also coming from an accredited program. So the integrity of the exam is key. So that's the reason why she's studying with this. And then giving design tips and talking about authenticity, fairness, flexibility and choice. Choice again, the fact that students matter in the process of really trying to kind of have something that is targeted to their specific needs, the aspect of trust, approachability and what informed. Because we don't have that much time today, we only have 35 minutes left. We've been saving you from going through a long list of items. So basically what we've done is identify four of them. So I think is the next slide. So we're going to spend a little bit of time in talking about integrity. So we're going to be discussing what we mean by integrity and then having two or three guiding questions to kind of help you think about, okay, how does, you know, integrity look like in my particular context. And then talking about design, we can also talk about students matter. So really having a learner standard approach in the way we design exams and talk about approachability and also what it means by that. Before we start, I just want to make sure that we have enough time at the end to adjust some of the questions that you talked about that you asked earlier. So please allow us to go through the slide and then we will have time for some discussion. So integrity, I would like to encourage that we talk about integrity throughout the term. So for Anita who is planning to teach in term two, so make sure you start talking about it earlier. But now if you're only thinking about it now because there's a final exam, it's not too late. Start having an open discussion with your students about what integrity means to us as a human being, as a professional. And so it's like a piggy-palm is a present of there are examples that we can follow before we do that using the slide to help me. We also need to make sure that what is allowed for our students to do during the final exam and what is not allowed. Because we all teach different courses, everyone has a slightly different definitions of integrity. Some of you are going to be doing open books, some of you is closed book, but what is open book and what is closed book? What is that one page of notes? How big is the page? How much information? You need to have a detailed description for what you consider to be allowed and what is not allowed for your course, for your students. Because each of us is different, we set different guidelines. So it's important to let the student know. And we know that integrity is important for all of our assignments, but for our final exam, this is how we are going to deal with it and explain it to the students. A very simple way of dealing with integrity for an exam or assessment is that we will put a pledge in the beginning of your quizzes, exam, and you will say, as a student at UBC, I solemnly pledge that I promise I'm not sharing. So there are different examples. People, there are some standards you can use from the link here. There's also engineering specific or anything that you will use in pharmacy or in education that you would like your future engineers, your pharmacists, your future teachers will be able to say to the future clients and the professions. Very simple way that could be a statement that you put at the beginning of your final exam. We've seen people who asked them to say, I agree with the statement above, yes or no, I agree or disagree. I always wanted to have that discussion. What if the students say disagree? You're basically forcing them to say agree. And I'm thinking, if I'm going to use it myself, I may actually ask them to do with you in the blank. A little bit of a customized, personalized, so then the students are actually going to think about that, not just, okay, I'll do it, but actually make them do something about it. So my question to you is, I wanted, because I'm going to change, I'm going to ask you the questions, how does it going to work in your profession? What does it mean when you decide your final exam and your challenge and concerns that you still may have? Okay. Feel free to type your questions and challenge when you are using this in an integrity patch. And I'm going to move on and have more time for discussion later, perhaps. Okay. Manuel. Oh, I thought we would be spending a bit more time in the integrity piece. So, Andrea, I just put the link in the chat regarding the final exam integrity page. So, the design aspect, so in this place, we just want to talk about, if you're planning, depending on what kind of exam you're planning is trying to have something that talks a bit more about the higher level skills that you'd like to measure and not necessarily the information that you expect students to recall. So trying to have something a bit more complex. So in this particular case, the Laura was kind of explaining, well, you know, having high level type of questions that require a little bit more thinking, a little bit more time instead of having, I would say, simple level multiple choice questions. So that was kind of a strategy that you really try to kind of, you know, share with us basically. So I don't know if you've heard, but we tend to refer to the Bloom's taxonomy when we try to kind of come up with higher level thinking type of questions. So, you know, you have six different levels of complexity, I would say. So if you look a little bit at the type of exam that you have, really think about, you know, if you could change some of the questions that you have and really trying to kind of measure very specific skills, maybe more complex skills, that sort of things. So there's an exams not necessary focusing on the recall of information, but really the application of knowledge maybe. Creating questions that mostly require students to use more than one piece of information. So trying to have them, again, look for resources, use a little bit more creative thinking in their approach, which is something that we could do. And the design, the reason why we emphasize on design is that sometime when you have something online, you may go a little bit higher than what you've been doing before. So based on that particular design aspect, think about, you know, how would this influence the design of your final exam? What are the things that you could potentially change, assuming some of you already have, let's say, a quiz that is, you know, having multiple choice questions. I have to, you know, what are the things that you could change in those multiple choice questions to kind of really measure something a bit more specific, maybe. And what could be the challenges or concerns, you know, taking this approach, you know, maybe a concern around the time that we would take to take, you know, to change some of those things or always make sure that, you know, you're still measuring what you're supposed to measure. So the second piece with the design and when we talk about assessment is always make sure there's a good alignment with your learning objectives. So I can imagine for, I think Andrea, you're coming from an accredited program, you always have to make sure that there's a clear alignment with your, I would say, program outcomes that, you know, you're measuring what you're supposed to measure. But I would say it's always good to kind of take more like a reflective approach and look back at what you currently have and what are the things that you could potentially change. So then it's not necessary just recall of information. So that's the reason why we really focused on design, on the aspect. So you could use the chat if you want to share a little bit about, you know, what you're planning to do in terms of design. But if I can give you a tip of advice, I would just say that takes time, you know, to rethink a little bit, you know, what you've done, or change some of the questions that you have or even the entire type of exam you've decided to do. Maybe you realize that's not necessarily a good fit for online. So there are a lot of things that fall under these design umbrella. That's for sure. Thank you, Judy. So I think we can move on to the next year. Student matters. Well, for final exams, it seems like a majority of participants today are interested in more security. But I think, and also you may focus on what is the most accurate way to assess the student knowledge, which is very important. But I think during this unprecedented time, pandemic situation, more importantly, you still need to know that you care about them, not only their success. So I think this is really important, educators to demonstrate professional caring. So be kind, pay attention, tell stories, invite stories, show you care, don't be afraid of love. This is a quote I saw from the article. So article is based upon the nursing courses. The author is a professional nurse and educator. So think about in what ways can you show that you care about students, your caring should be presented throughout the journey of the course. But especially for the final exams, what can you do? And I think there are some ways to do, you may want to pay more attention to student needs, actively listen to student needs. And also you may think about alternative assessment these days. And you may want to ask a student to feedback about their exam experience and challenges. And also I strongly encourage you to reflect on your own experience with care. We unconsciously care for others in the way we have been cared for. So I think a caring relationship with the teachers help a student to do better in learning. Yeah, we can move on to the next one, Judy. I think go back, back one. There we go. So I think this relates a lot to what Suna was just talking about. But when you're setting up your final exam, it's important to have a plan for how you will communicate with students. And think about not just, you know, you communicating ahead of time what the expectations are, which is really important to be more clear, more clear than normal, just because students will have anxiety about doing an exam online. And it's important for them to understand what to expect. There's a lot to deal with with the technology setup, especially if you're using any type of proctoring tools that students can get anxious about that. And so just being very upfront about the expectations can be really helpful. But also during the exam and then after it's completed. So no matter how much preparation you have, there's always bound to be some type of technical glitch or someone's wi-fi as problems or a computer issue, something can go wrong during the exam. And it's important to have a space that you can use that students are welcoming, told about ahead of time where students can contact you during the exam and you can deal with those issues as they arise and that you have a procedure for that. So I mean, some strategies that instructors have used is like to set up a room in Zoom where students can go in there and you'll have a TA or yourself in there that you can help troubleshoot problems or giving them contact information in email or a phone number and having a strategy for how that will work. And then also becoming, if you're doing any type of timed exam, becoming familiar with the tools in Canvas for how you could make small adjustments to a student's exam and see the history of what they've done so you can monitor those as they happen. But another thing that's just very good to do upfront is to engage students in a conversation about it, about the exam and find out what are the issues that they're concerned about. And if you've done midterm exams, get their feedback on how that's gone for them, what have been their challenges, what they've liked, what they haven't liked. And you can use that information to help you. So I had a conversation with Matt Yedlin who's a professor in electrical engineering the other day and he was telling me about in his course what he did is he had two midterm exams, so two midterms and a final, and he said after the first midterm exam, he created a discussion for him in the course and opened it up immediately after the exam and he described it as a pressure valve release, that students got out of the exam and there's a lot of emotions and frustrations and nervousness about how they did and just had the students share what their experience was in there. And then students expressed a lot of concern about certain elements and they were able to address a lot of those for the second midterm and then he said it went much better but they still were able to get some constructive feedback. And so he said he feels a lot better going into the final exam because they've been engaged with the students about this and then it's also less in the anxiety of the students because they feel like they've had more agency in contributing to how the course is going and that their concerns are being heard. Okay effective techniques to assess the student learning online. Well I think this is more like full thoughts. So today I think many of you are interested in security pieces rather than more like a higher level thinking about you know what we can do during this pandemic. But I think to me learning takes place in students has where it is invisible to others. This means that learning must be assessed through performance. So what students can do with their learning? Consider various ways to assess the student performance. So former, informer, high-low-stake, anonymous or public individual collective. Hopefully you can think about multiple choice questions are so effective because you will get the result immediately. But I think during this time of the year during this pandemic situation you may think about some equity and inclusion aspect as well. If you have a final exam requires a thumb paper, open book essay, portfolio or group presentation, create rubrics to communicate expectations and reduce marking burden. And rubric is an effective assessment scoring tool that articulates expectations for the assignment or for the assessment. So I don't know have you ever used rubrics for your exams like a final project. So and determine instead of feedback strategies, instructor feedback or peer feedback to nurture student learning. I know even after final exam if you find there are specific areas that many students didn't do well you can still highlight and explain them through, for example, Canvas announcement to reinforce the student learning even if the course is almost over. Or if your final exam requires a thumb paper, you can encourage your student to use peer scholar in Canvas as an example. So for peer review, peer feedback and student can incorporate the feedback into their final version. So peer feedback plays a critical role in student development, motivation, satisfaction. So to promote the student learning, you may need to consider content feedback, amount of feedback, timing of feedback and revision opportunities. The next item I was thinking about was consider where equitable grading assessment can be added or integrated. Many faculty members believe that your assessment practices are fair and objective, but students learning can be easily disrupted by various reasons these days. So many students may have their own challenges, problems. So equitable assessment is an important, important, particularly during this time. One of the ideas would be if you have any plan to have portfolio as a final product, as a final project, then self-assessment can be part of the portfolio that will help the student to reflect their own learning and progress. And I think that involving students in the assessment process is a great idea. I believe there are a lot of techniques and strategies to assess the student learning out there. I encourage you to visit CTLT online teaching program, OTP module, particularly module three is about designing online assessment. And we have some key considerations. Yeah, so I think we've touched on a lot of these earlier in the approachability and focus on student section, but being very clear to students about what you want, what the rules are, what the intent is, and what your goals are for the final exam and then engaging them in conversation about it. So if you're using particular techniques that it can be helpful to get their get their feedback, let them know up front and answer questions ahead of time, give them the opportunity to do that, to engage in conversation with you and address those concerns before the exam starts. And then also, like I said previously, providing a space during the exam where they can reach you and especially if you're using things like proctorio and or lockdown browser that if students run into technical problems getting those to work, they need a space to contact you during the exam. And also, again, especially if you're using any type of additional software that adds to the complexity, setting up practice exams for students ahead of time, that they can use those, try them out in a low stakes situation with enough time that then they can deal with any issues that they have and let you know what those are. And I would urge you to check with your different faculty support units because I know many of the different faculties have set up tools already that you can use. So I know like in the Faculty of Arts, there's a site set up that you can a site that you can just send students to and they log in and then it gives them the opportunity to practice with all the different invigilation tools they might run into. Yeah, I think it's relevant to what Jason said and also chat between Manuel and Andrea. Yes, more detailed information, instruction, communication, clear communication with students will help a student to be successful in their final exam. So post-tech support info, if they have any issues during the exam, they need to go, they need to know where they need to go, right? So creating online exam support room, so sort of case as two tiers exam support teams. One is a graduate student help the invigilation after having some training. And next to tier two, support to staff, just to support a particular proctorial and lockdown browser issues, more difficult technical support. So if a student has any problems at the beginning or during the exam, they know where to go, just a headline. So tech support info should be posted in advance and also post to specific rules that student are allowed or not allowed to use during the exam. For example, in in-person class, during the invigilation, usually they didn't allow their calculators, but within this environment, they are allowed to use, students are allowed to use a calculator and they can have some snack even, but they are not supposed to have a conversation with their peers until until exam finishes. So certain rules should be posted so that students are allowed to know, are aware of the know what rules are they're supposed to keep. An informed student of general technical tips for the exam using helpline is important, but I think they need more general technical tips as well. So they should have reliable internet connection they need to if possible sit beside two routers and start to their laptop if they have any problems at the beginning and finally contact the helpline, etc. So as I mentioned before, overall instructor's clear instruction information and communication with your student will help the student to be successful in their final exam. Any questions? And I need to follow the chat. Before we ask questions, we just want to post this because I think this is one of us put this slide at the end. The primary purpose of assessment is to improve students' learning. And I think we need to continue to keep this in our head as we develop our final exam.