 So hello and welcome to everybody who's coming in. Nice to see you all. See some familiar names and faces. So welcome everybody. Okay, so welcome everyone. We'll get started since it is now 3-0-1 on my clock, but welcome, really nice to see you all and kind of congratulations for hanging in there at sort of the end of the day and appreciate your persistence and still being able to kind of show up for something like this. So really appreciate you coming. Thank you very much. So I'm Candice Rideaud. I'm an associate professor of teaching in the faculty of land and food systems. And you can see Gabriel who is one of the learning analysts in our LFS Learning Center and will be the people that are facilitating the session for you today. We're going to be sharing it sort of in a half and halfway. I'm going to be sharing the sort of pedagogical thinking and the practicality of implementing a flexible assessment approach and what its impact has been on my students. And Gabriel will be speaking to an app that he and his team have created that can integrate the logistics of implementing flexible assessment directly into Canvas to make things much more streamlined and accessible for people. So that's how we'll be sharing things. But to get us started, I want to acknowledge of course that so much of the work that we'll be sharing today was done on the traditional ancestral and unceded territories of the Musqueam people as we were based at UBC. Today, I'm joining you from the traditional and ancestral unceded territories of the Squamish, Stolo and Slalo-tooth people and just joining with humility and gratitude for the stewardship of the land and the opportunity to be here and really kind of reflect on how the way we do our work can be an extension of that respect and humility. And part of what we're talking about with flexible assessment today is really looking at ways that we can design courses to make them more accessible for diverse learners with different priorities at different times in their life and really being able to support a personal and meaningful engagement with a course. So that is kind of what our aim is going to be and how we're going to approach things. And as I said, I'll sort of be speaking at first, but please don't hesitate to let me know if you have any questions. Initially, we will also have time for open questions and discussion at the end of the presentation too. So from theory to practice, how it is that we can actually be streamlining and implementing flexible assessment within Canvas. So as somebody who created a flexible assessment approach more than 10 years ago now, when I joined UBC as a faculty member in 2012, I was presented with kind of an interesting logistical dilemma in terms of how I would align my own teaching philosophy and my priorities for students in my courses with some of the existing structures that I had to work within. So I was teaching very large classes. And as some of you may know, when you're teaching a class, you have a lot of latitude to figure out how you will approach it. For one of my courses, a particular constraint I had was that there was another professor teaching another section of the course who had been for quite some time and had quite a traditional assessment method using basically a midterm and a final exam. And I knew that I wanted to do a little bit more than that, but also needed to keep the course quote unquote equivalent to what would be happening in the other sections. So with that kind of logistical constraint, I was really reflecting on what is important to me in teaching. And here kind of showing just a snapshot of my pre-pandemic, one of my classrooms and the many, many students in those classes. I was really thinking that ultimately the most important thing I think that I'm trying to achieve is how we can consistently create effective learning environments. And I can't tell you how many different words I'm coming up with other than effective, but ended up thinking, well, ultimately it is effective, but we want things that are transformational, impactful, motivational, inspirational, all of those sorts of things. So trying to combine the hopes that I had with the logistical constraints that I was trying to operate within led me to develop this flexible assessment approach that we'll share in today's session. Now first, before we kind of jump in, I wanted to ask a quick question of you. And if all goes well, you should be able to actually see this as a poll on your slide or on your screen. Just to give a sense for Gabriel and I in terms of how we would like to explain things, what we might dwell on or provide just a brief overview of, I'm wondering if you can indicate what brings you here today in terms of the role that you fulfill when it comes to educating students. So you can see a variety of options there, some perhaps related to teaching your own courses and not having used flexibility and assessment or teaching. And you have used a form of flexible assessment previously. Do you support others through educational design and development or are you working in IT and kind of really interested in the technological end of things? Okay, so we have almost everybody has responded. There we go. And luckily I didn't know, did I anticipate all the categories? And nobody ended up needing to choose that none of those options share them or suit them rather. So you should be able to see on the screen just for your own information that the majority of you about 60% do design and teach your own courses an additional 17% do and have used some flexible assessment already about 20% supporting people through educational development and about 13% working in IT. So thank you, folks. That helps us to just kind of understand and know how we can best communicate things. So thinking of that, I don't know, dilemma just came to mind. How do I marry the ambitious hopes that I have with the logistical constraints? As somebody who is now a faculty member in educational leadership in that stream where teaching is super important but comes from a background of sort of human based behavioral type of research. It was kind of natural for me to think of, well, what do we know about the theory of what can actually motivate students and how can we alter our learning environments? And one of the theories that I was familiar with from my work in health related contexts is social cognitive theory, which is very all-encompassing, very complicated, has a lot of things to do with it. But at its heart, it's really talking about the interactions of various factors, determining the behaviors that people undertake and that learning that's occurring in our courses is affected by these factors. So the behavior of learning is going to be affected by the individual characteristics that a person brings to that situation and importantly by the environment. And it's the environment that we can create and affect our classroom environment, our course design. And it was really clear to me that one of the things that I think most drives student experience is what we have them do for marks. So basically I remember a long time ago coming across the phrase that assessment drives learning. And I really do find that that has been my experience in terms of my own observation of student behavior. It's what we have them do for marks that can really determine what it is that they will end up learning. So thinking, okay, how do I match up an existing section of a course with what I'd like to do with it? And I ended up coming up with the idea of creating essentially a proposed grade distribution, which is sort of what I'd really like to do that deviated from the midterm final of the other context. And then at the time, providing options that would enable people to kind of end up with that type of assessment, if that's what they wanted. Now over the years, it has developed further so that what you see here still doesn't end up with exactly just a midterm final. What you see here is actually what I'm doing this particular term. I happen to be teaching a large third year course in this term. And you can see there's a proposed grade value that would be what students would use if they don't specify otherwise. But I do provide them with the option to make some adjustments within ranges that I specified. So it's not a free for all. It does need to be selected from within these ranges. And importantly, the decision students make about what they will do for their work in the course is made at the beginning of the term. Usually about a day or two after the ad deadline for a course is when I set the deadline for them to submit their request. And so it's within about the first two weeks of the term or so that they would have the opportunity and the responsibility to decide if they would like to make a change. They can use the proposed grade value by default or if they want to adjust it depending on their own priorities, their interest in learning, the things they might like to do, they can make a change at the beginning of the term and it becomes a learning contract. This is the plan. This is what's gonna happen. It's not something that changes later in the term other than if there were normal accommodations we would make for students in crisis and other things that you would make in the absence of flexible assessment for sure. We still look at those things but it doesn't lend itself to changing throughout the term for other reasons. And my reasons for doing this, I had sort of mentioned in terms of the situation I found myself in, but I was really hoping that this would increase students' motivation and engagement actually make them learn more because they had selected the things they were going to do and hopefully also enhance their well-being by not overloading students all the time and expecting that everybody needs to do the exact same thing. So these were kind of my key motivators but I was also really affected by the logistical constraints I mentioned and also the hope that this would help me to conserve some resources in terms of TA marking support and such where we might not be able to do some really interesting in-depth assignments if everybody was doing them. But if it's a subset of the class, then perhaps we can provide the adequate support and marking and feedback and those types of things. So I've subsequently realized that there's a number of ways that we can implement flexible assessment. So what I will share more of and have outlined is the approach that I came up with but there's other things that maybe some of you are already doing and certainly other things that I could find by looking at the literature. One of the most common is what they call grade dropping just letting students drop their lowest score on a series of quizzes or assignments, whatever the case may be. This has some merit, it certainly lends itself to easy management but it has some significant drawbacks because depending on the particular context it has been shown to reduce the effort that students will put into a particular test or assessment or a quiz because they always think, oh, if I don't do too well, I can just drop this one. So in terms of whether or not it actually enhances learning, I'm not sure if it hits the mark on that one. Another approach that has been shown in the literature is to assess the values of particular assessments at the end of the term to optimize a grade. So for example, the type of thing where students would know depending on what serves you best one of the following will be done. It might be that your final exam if that score is higher than everything else it will be your final score in the course or it could be a portion and these other things end up contributing. So students know that, okay, this is what my options will be and just whichever one ends up in the highest grade will be the one that's used. But there's drawbacks to this as well especially when a final exam can end up being 100% of the final grade. And it's been shown to lead to delayed effort. Students don't necessarily aim to keep up and fully invest from the outset. So again, I didn't think this was really hitting the mark in terms of learning. So to my idea of having students select within ranges I found some examples of this where students would put together almost like a jigsaw puzzle what will they do at what value? And they were given sort of full possibilities on various items. And every student we need to figure out okay for a class participation and such I'll choose this and for the case studies I'll choose this and just make sure it adds up to 100%. This is getting closer to what I thought would be useful for personalizing a learning experience but it seemed kind of overwhelming and stressful even to me nevermind to a student who maybe just hadn't had the option to do this before and might be thinking I don't know how am I supposed to know which one I'd like to do and which I should value more. So where I have landed with providing the proposed grade distribution eases some of that stress while also still enabling the flexibility of making priority choices depending on what people are interested in. So the proposed grade distribution I make clear to students is what I would do if I wasn't offering flexible assessment that I believe it'll meet the needs and promote the learning of the vast majority of students. And so if they're overwhelmed don't know what to do or even just think yeah that actually looks pretty good to me they can just use the proposed grade distribution and it's kind of easier they don't have to make that decision at the outset within that first about two week time period they can make a change as long as everything falls within the range for that item and the total equals a hundred. And you can see here this is kind of I don't know maybe about five years into the amount of time I'd been using it at this point in time I had one element that wasn't included in the proposed grade distribution that students could opt into. The reason why that wasn't included was actually just again logistics I couldn't do it for everyone but I could do it for a select number it's seven D students that I can accommodate in a hot topics discussion. So I had that as a possibility for people to opt in but over time, especially quite frankly since the pandemic altered our approach to teaching with the remote transition and such this has also evolved to match the circumstances that we find ourselves in. So I now have an additional assignment that's not included in the proposed grade distribution something that I think might be pushing some students boundaries a little bit more than they would care to be pushed in this particular context because it's something that promotes them or provides the opportunity to create art for social justice and do an analysis of the issue they're trying to communicate through their art. Some students blossom and are so grateful to have this opportunity. And I'm certain that most students because it's not the majority that select that are probably grateful that they're not sort of being forced to do something like that. But there's a few things that they can choose to do but that would mean having to make some kind of assessment change in the other values. So this is sort of where I've landed this is literally what's happening this current term. And as you all think about, well, what might work for you or like how do we think through this type of process? It's these questions that I think are the most important considerations. The first in terms of thinking, what is your own aim in terms of introducing some flexibility into your own course assessment scheme or the assessment scheme of a faculty member whom you might be supporting would be really to think of what are you trying to achieve because we have to have a clear idea of what the outcome is that we're trying to accomplish what our target is. That will have all of that kind of trickle down impact on the decisions that we make. And I spoke to mine in terms of the impact I was hoping to have on students and initially the opportunity for equivalency with another course section that was necessary. But these types of questions how much should each item be worth? And is it maybe that something's even worth a zero and students can opt out of it entirely? And if so, how do you make sure that they actually achieve the course learning outcomes? These things are really important considerations and in a really fun way, it's kind of almost like a game of figuring out, okay, like what part is essential and what is assessing everything and making sure that with the minimum, quote unquote, minimum being selected where would people end up in terms of the learning outcomes? So it's a really informative process but it needs to be really intentional when we think what might these ranges be? Over time, again, specifically with that transition to remote learning I've had more things that students can opt out of entirely but the math doesn't work to enable people to only do the final exam and the other essential item even if they had those at maximum value it would only be 75%. They still need to make their selections which is another way we can kind of influence the choices is how does the math work out and what is necessary to get to a hundred percent? When we're taking this type of approach especially having students decide at the beginning of the term which I think is essential both for students clarity, reducing their stress and to avoid that potential pitfall of delayed effort that the other strategies I mentioned from the literature have to enable students to make a choice at the outset they do need to be able to make an informed decision. And so here these are just screenshots of the instructions for the various assignment type assessments that students may select within the course that I teach one of the courses that I teach. So it's important that Canvas has all of the information they need in order to figure out what might be a good fit for them. So the instructions are there, the rubrics are there all of those sorts of things so that the students can make a good informed choice. And even to some of the other things like I have a series of readings that there's a quiz based on each of the readings that lasts an hour, it's an open book quiz intended to support their learning but it is assessed certainly important in terms of assessment as well as learning. So for those I actually provide some examples of past quizzes that I am no longer using the readings have changed but I'll provide the sample reading and the sample quiz so they can see all those things at the outset. And at this point I just wanted to check in and see what you were thinking because I always find it interesting to reflect on what we think might happen and then what actually does happen. So one thing I'm wondering what your thoughts are on this is if you take a look at the quizzes based on readings this is the one for this current term. You can see I suggest they be worth 20% of the final mark there's 10 quizzes that would be 2% per quiz but students can opt out of those quizzes. They are very regular and they can only be completed in the week that they're available. So they can't stockpile the quizzes and get through them all at one time. It forces them to be keeping up with the material on an ongoing basis. And I'm curious when you think about this and what you think might happen what proportion of students do you think might opt out of the quizzes because it's kind of too much consistent work throughout the term like two quizzes a week in the summer term in the regular term it is just one quiz per week. So I've put another pull up just to see where your thoughts are at this point in time. So what proportion of students do you think would opt out of non-essential meaning it could be zero. So non-essential activities such as the readings quizzes just using that one as an example. So I'll give you a moment to consider and I'll share as you're entering your responses that I thought it might be a bit high and maybe that reflects how I would have approached it as a student thinking like, oh my gosh do you want me to do 10 quizzes for 20% I think not if I have the option. But yeah, so thinking it through I just thought I wonder if they'll sort of want to put more emphasis on things like the midterm and final exam in terms of assessments or focus more heavily on the assignments. Okay, so we've got pretty much everybody has responded to this. So I'll show you what the distribution looks like and it's interesting because with the exception of the 76 to 85% we have some guesses all along the way probably the most common is guessing that five to 15% might opt out but actually equally so is 66 to 75. So on the flip side maybe three quarters will opt out or less than a quarter will opt out. And I think this reflects kind of both my initial uncertainty but in also my suspicion that it might be a fair number. The thing that has forever surprised me and I offer it as potential surprise for you as well is the wisdom of the choices that students make because hardly anyone opts out of the quizzes based on readings. The lowest number completing it that I've ever had is 95% meaning that 5% opt out. More commonly it's even less. So in this summer term where there's two quizzes per week they're real readings like you have to work on these even though it's open book and they're not meant to be stressful but they're meant to be kind of a deep dive into the reading. It's 1.2% of students. It's three out of 244 students have opted out of the quizzes based on readings. I find that impressive in a way that reassures me that students are being thoughtful about the choices they're making and I consistently see that they're making the choices they think will lead to better outcomes for them. So better learning, better grade, better overall experience and that plays out in the choices that they make. So moving from kind of how I've set things up to a little bit in terms of what I observe and what students comment in terms of the impact. In terms of the choices that students make it hovers around 40% of most sections of the courses I teach using the proposed grade distribution and 60% or so making a change. Which for me is really strong evidence of the importance of having a proposed kind of default grade distribution and not just having everybody make a change. It also shows the importance of enabling change because for 60% approximately something else would work better in terms of their commitment to their learning. So it hovers around 40, 60 but just to give you a little bit of a sense in term two that just ended in April, 42% used the proposed grade distribution and the course that I'm teaching this summer it's 37%. So again, sort of on either side of that 40 with a few more making some changes in the summer which has the additional constraints of being a much quicker course progression. It's kind of much more intense because we're doing more in a shorter period of time. But I consistently see that the majority of students are completing these non-essential assignments. They're not trying to do as little as possible. So assignment A, that's the one that I've had all along. It prompts them to take action to help end world hunger. This is in the context of an international nutrition course. And somewhere between about two thirds and around 80% of students consistently select that assignment. And in fact, it's been increasing. It's part of the proposed grade distribution. Some students increase its value, some decrease it but still do it. And I must say in terms of the logistics and trying to plan for TA time, it's both with encouragement and again being impressed that our students are agents for change and choosing to do something like this that I see high buy in but with a little bit of dismay oh my gosh, how are we gonna mark all of this? Because this term for example, we have I think 195 of 244 students. It's just over 80% of students are doing that assignment. So sometimes it's a bit higher than I expect because in the summer I actually thought it would be less. We have as I said between 95 and 99% completing those readings quizzes. And the other things that people can opt into the hot topics discussion which is kind of limited by the capacity but it's between 20 and 35% select one of those and about 12 to 15% have consistently opted to do the art for social justice assignment. So we see a variety of different types of choices. Some people really try to make their exam type assessments worth as little as possible and they bump up the assignments. Some go in the other direction. There's almost no very consistent patterns that we see. People really do personalize things. So a little bit in terms of a student's perspective on this flexible assessment approach. One of the sources of insight I have on this are mid-course feedback surveys that I consistently do. I'm a little kind of quantitative in my approach. There's options and capacity for students to give comments on this but I know that the even this is just a minor screenshot of some of the questions I've asked might be more than would suit others but I do ask very specific questions and have them kind of enter their response on this kind of level. And I thought it'd be interesting to share some differences I see from analysis I had done for two years in the sort of pre-pandemic period and then analysis that I had completed in the courses since the fall of 2020. Because pre-pandemic I've got over 500 students about 30% of the classes had completed the mid-course feedback survey at that point. Post-pandemic there's about 336 about 25% of the ones that have taught the six courses I'm drawing these data from since then. And these are the questions on this screenshot that you can see that are in common where I'm asking about their impressions of flexible assessments. So are they glad they had the option and did thinking about the final grade and how it would be calculated affect their engagement in their learning in the course. Now pre-pandemic this is when everything was face to face and such. In response to that question I'm glad students had the option of adjusting how their final mark would be calculated about 86% agreed or strongly agreed and only about 2% disagreed with that statement. But interestingly in the time since fall of 2020 I've continued to teach this class remotely since that time the agreement and being glad has increased. It's not actually 98% are agreeing or strongly agreeing with that statement. And there's actually none who have disagreed or strongly disagreed. From the more pedagogical perspective thinking has this made them more engaged in their learning in the course which requires a bit of metacognition to even reflect on and answer that question. Prior to the pandemic it was about two thirds felt that yes it had strongly agreeing or agreeing and about 8% felt that no, it hadn't this didn't play a role. Since the pandemic about 81% feel that yes this has really made a difference in that regard and about 3.5% feel that it didn't. So from these initial bits I can see it looks like people are glad to have it and it is impacting their perception with their engagement with the course. I also have a little bit of focus group insight from comments students made in small focus groups that I've done. So I'll share just a couple of the comments here before passing it over to Gabriel to speak to the mechanics of how this can be integrated into Canvas. But students have commented that having this capacity to choose this flexible assessment approach makes them feel more empowered. An example of that comment it makes you feel like you're in control of your own learning and they've rarely had that experience before in any of the classes they've taken. So it was something really different and pleasant and enjoyable. It makes them feel respected as learners. So the student observing I don't know how to put this into words but I think it made us feel kind of important. You have a say in how you get graded and your opinions matter and what you have to say matters. And I think that gave the whole class environment this positive uplifting vibe and thinking how our assessment framework influences the learning environment we create thinking back to the social cognitive theory and the interactions and those factors. It also makes students feel accountable. So saying, I think choosing the weight of your grading gives you accountability. It makes you work for it because it's something that you chose. An additional example of that was I think the responsibility was a little like, whoa, I get to be accountable for my own grades. Can also see that it directs students' efforts. So thinking here about the quizzes based on readings observing, I think if I had quizzes in another class where I didn't get the chance to choose the percentage I feel like I wouldn't have put in the same effort even though they're worth the same percentage in the calculation of the final grade. But I think the accountability and responsibility behind purposely choosing to do an assignment pushes us to make more effort. So a lot of really good things being observed but one notable consideration again is stress because choice can be stressful. And this was stressful for some students but not for others so it depends and anything we can do to make it less stressful is something to keep in mind. So here showing potentially stressful I think what kind of like scared me was that there was no way we could change it after we made the decision. It does make sense but it was still nice to have the option at the beginning. So kind of saying like, oh my gosh I'm making quite a commitment. And then on the flip side somebody saying I felt less stressed knowing I could allocate my time to the assignments I would know I know I would do better in. So just different factors of course very diverse learners and different people. So I've outlined the approach, the reasons for it some examples of what it could look like and the impact it appears to have on students and their learning experience. One of the things people are often wondering though is how to do it logistically. I myself handle it through email up until this point in time but in the past year Gabriel has with his team developed an app that can be integrated into Canvas to streamline this whole approach. So at this point I'll pass it over to Gabriel who can speak to the idea of the mechanics of how might students communicate their choice to you. And I'm thinking of course by when but you'll see that's something you can put into the app as well. So I'll stop my screen sharing here and give Gabriel a chance to bring up his screen and don't hesitate to let us know if you have any questions and we'll have time at the very end as well. Okay, is it visible? Yes. Okay, so thank you Candace. So hi, my name's Gabriel and I work for the LFS Learning Center and Candace has talked about the theory behind her flexible assessment approach and now I'm gonna talk about how we created an app for Canvas that will facilitate that process. So the question is why should we create an app for Canvas and how did this project get started? So instructors in LFS have been using Candace's flexible assessment approach for quite a few years now and I've seen Candace presented a few times but one thing that I noticed was that there was always a few who were hesitant and they felt like it was gonna take up too much time. So one day one of my colleagues, Adri, and decided to document the process and the document ended up being seven pages so that was a bit much. And so even, but even though it doesn't actually take up that much time there was a perception that it would. And then we were on a team meeting brainstorming ideas as we often do. And one of my managers, Edmund said we should create something for Canvas and my other manager, Cyprian was always saying we should do something that no one else has ever done before. And we're as an instructional support unit in a faculty we are somewhat limited in scope by what our instructors in our faculty are doing. So for example, you can't just go off and create a dating app for Land and Food Systems because it has nothing to do with Land and Food Systems. So I thought, okay, so we wanna create something. What's something unique that LFS instructors are doing and it is the flexible assessment approach. And so I put the ideas together and we decided to apply for a TLEF to make an app for Canvas that would make the flexible assessment approach a little bit easier to do. And we got it. So let's look at the processes first. Process that we're trying to streamline. So in the original process there was a few different ways that instructors were doing it. So oftentimes people would create a survey to collect the weights from the students or some were using email. So the students were emailing them and then they were putting it into Excel. And then once the assessments were completed and all the grading was done they would export from the gradebook in Canvas and apply the weights using an Excel formula and then import it into the FSC. And yeah, I have import final grades into Canvas gradebook but I think most people don't do that because it's just, I think it's a little bit difficult. So you can notice that there's like a few different applications involved here. And so what we wanted to do was streamline this so that you're just using one tool to do everything. And yeah, so we decided to consolidate everything to Canvas and we definitely didn't want to create like one more system outside of Canvas because there's already too many of those to deal with and let's face it, most of the course activity is going on in Canvas. So there's a general preference I think for things that are integrated with it and it just makes everything a lot easier and more convenient. So I'm gonna go over to the live demonstration and I'm gonna show you exactly what it looks like. Okay, so here's my Canvas dashboard. I'm gonna go into a completely new course that has never used this before. So again, new course hasn't used it. So I go to the settings page and like a lot of tools, you go into the navigation section and you enable it. So here it is flexible assessment, enable and then you save it and hope that it actually saves. So here it is. So I click on it and it takes me to some instructions for how to set everything up. There's a bit here just to tell you how to use it. So I go to this course setup page and this is where I can input my assessments. So for example, let's say you have quizzes and default. So if they don't choose one, if they decide not to choose any weights, this is the recommended percentage. And then the minimum and maximum that you will let them give to this assessment. Okay, so let's do just a basic course. So you have an assignment. So let's say the default will be 20 for this one. Don't want them to opt out of it. So do minimum 10, maybe maximum 40. Again, I go to midterm and I guess I'll do 30 for this one. And minimum 20, maximum 40. And let's do final exam. So again, you'll notice it kind of turns green over here just to indicate that you've successfully done it. And let's do 40 again. And here's where I can enter the availability date. So this is the first date that they'll be able to enter. So let's say today, and let's say the add drop is the 10th. So we'll make the add drop into the last date by which they can enter their desired weights here. So 11.59 and I don't recommend 11.59 because if something doesn't work then you don't want emails at midnight. But anyway, here it is. So that's what it looks like from the instructor's perspective. So now as a student, and this does work with the student view, I go in and I click flexible assessment and I enter my desired weight. So let's get crazy here. Enter something outside of what they're supposed to do. Okay. And then agree that your grades will be calculated. You can enter a comment and submit. But oh, I didn't do it right because the maximum is 20 and I entered 30. So go down. And I'll make the final exam. It has to be 30. My addition is very good here. Okay. And then submit. And they can go back and change it before the date. If they decide to change something before the close date there. So there's the availability down here. So that's what it looks like in the setup phase. And now I'm going to go to a course that has some grades in it. So this course, all of the grading is done. Oh, I'm in the student view. Ha ha. Okay, so I'm in the grade book as an instructor. And you can see I've got all the grades done here. And I go to flexible assessment. And the front page gives me like a little bit of analytics here. So you can see the defaults here on this side. And then you can see what the students actually did. So that's kind of interesting. We might actually change this a bit because it's hard to see these. We've found if you have like a lot of assessments. So there's a few little things that we're still like working out the kinks. I mean, it's completely new software. So it's not perfect. And let's see here. So if I go to student choices it gives me the choices that my students chose. So, and it shows me their comments as well. So you can see Jane Doe here that decided to make her quizzes worth the same as her midterm and final exam was worth 40 and opted out of the assignment because she likes quizzes. She wants to do more tests. And then Dimitri over here decided to make everything equal except the final exam. And actually in this case the final exam was not flexible. So the final exam is worth 40% regardless. So they could not actually change that. So that's just kind of an example of what students might do. And then once the grades are in you go to final grades. And the way that it calculates this is through the assignment groups. So in assignments you have these categories that you can create for your graded items. And that's how it tells what the grades are. And that's how it calculates the grades. So I just, yeah, you do have to set up your grade book in a certain way. So there is that. But once you're done, you're done. And now it pulls all of the grades and it tells me what the totals are and it tells me what the default would have been and the difference. So it's actually kind of interesting here that the difference is not actually that much. And then I get some averages down here. I can see all the different grades that the students got. And then I can also export this. So if I don't want to use the Canvas grade book to release the final grades I can actually just export this. And then you can check the grades and make sure that they're all calculated correctly. And then you can import them into the SIS or FSC rather. So if you want to use the Canvas grade book you can then do submit grades to Canvas. And it will ask you if you're sure. And now when I go back to the grades I will see those grades in the override column here. And that's how it works. So I'll go back to my PowerPoint because I have a few little closers here. So yeah, future plans. So it's not done yet. Right now we have one instructor using it in philosophy actually. And- Thank you. So can I just pop in the one section and just be sharing screen right now because we can't see your screen anymore. Oh, sorry, I stopped sharing, didn't I? Okay, I'm sorry. Thanks for interrupting that. Share. There we go. Okay, so these are some of our little future plans. So we have one instructor using it in philosophy right now. Another instructor in LFS, Judy Chan will be using it for the S2 term. Some things that we want to add before then are some customizable instructions for students. So you can have kind of like a welcome message at the top of that page where they're entering their desired weights. Maybe some little clearer error messages for students. There's a few little bugs. Like if they go to the page for entering their percentages before the open date, it says, it makes it kind of look like they've chosen the defaults already. But so there's a little bit, like there's a little small things like that that will be fixed. And beyond S2 term, we're hoping to pilot this with more instructors outside of LFS and eventually have a university-wide rollout. So this system is available only for courses under the faculty of land and food systems. So any LFS instructor, and I see there's a bunch here, you can actually go in and start using this now. And another thing that I wanted to point out is just like the ongoing audits and maintenance. So when Canvas does an update, we will have to test this to make sure that it still works because there's always a possibility that Canvas could change something, there could be a bug or something and we wanna make sure that it's working. And a lot of times people have these projects and they don't think about the maintenance involved. And we actually do have like a full-time developer in the learning center and we have two student developers as well. So we do have that ongoing support that I think some others don't. So yeah, if you have questions or you're interested in piloting this software, you can contact me at g.smith at ubc.ca. If you contact me at gabriel.smith, you will contact Gabriel Smith from ARC, which is not me. So that's all I have. And let's go over to the questions. So feel free, folks. If anyone has any questions that you'd like to either put into the chat or feel free to unmute and ask a question, we're happy to chat and answer any questions that you might have. Stephen, go ahead. Yeah, I have already a giant mental list of things I'd like to see in that app for Gabriel, but I'm in a different department on another campus, so I'm not gonna make contact. It won't come to me for a while. I did have a question for Canvas and it looked like there was information on the app that might have allowed you to do this. Have you looked to see what proportion of students are actually making good choices, which is to say that the adjustments that they've made to their grades give them a higher grade than the default setting would have done? And do you or have you considered offering them a fallback position of if you make a bad choice, don't worry, you'll get the default if you wound up choosing things that we're gonna actually pull your grade below that? So the answer to the first question is yes, that I have looked, though the last I looked was ending in, I think, maybe 2017 data. There's actually a paper that I published. So if you kind of Google, yeah, yeah, okay. And it'll come up. And I haven't done it since, which I'd like to do. I kind of do an ad hoc analysis, just rough and dirty at the end of each term, just to see anything happening that I need to know about. And essentially what I see is not large difference to the hypothetical example that Gabriel came up with. It's almost similar to that. Like it's a little bit of noise here and there, but nothing that kind of strongly shows, oh, that pattern of choices results in a higher score. The thing that I'm very aware of, though, is how almost impossible it would be to evaluate some of that information because there's so many confounding variables. Like students that want to emphasize assignments and less so on exams are often students that perceive themselves to struggle with exams and examining viety. And so there's so many factors that affect performance that go beyond just the choices that they're making in this case. Sort of the anecdotally and the things I can infer leads me to believe that most students are doing better than they would if there hadn't been the option for flexible assessment. So, but whether they're doing as well as they would have, you know, with a different choice, not only do I like beyond that initial assessment that I did, I don't look at that regularly, but no, I don't have any plan whatsoever to offer a backup of, oh, if it turns out it would have been higher, no worries, we'll do this. I'm much more, you make a thoughtful choice. It's a learning contract. Forward, this is what we do. We invest in the learning. We have like such good experience, like all of those things, but no, like, oh, you know, like, could it have been 2% higher or something else? Because I am kind of fundamentally, you know, not really a fan of people's over-identification with the precise number that their grade may be. And so that would just kind of be, you know, entrenching an approach that students have that I tried to almost push back against. But the thing that I find interesting to note is that even term on term, I look to see who's scoring the highest, who's scoring the lowest, and just what other things can I see? And it's not like, oh, it's everybody who opts out of an assignment that ends up with a higher grade. In the initial analysis I did in 2018, there was a little bit of a difference, like two to 3% on those kinds of choices, but that hasn't really persisted. Like it's kind of been always sort of in the range. Last term, the person with the highest score actually had the proposed grade distribution, but I have had it be students with the highest score are those that know they do well on exams and like they focus on that. I have a lot of confidence in the fact that my final exam is a cumulative final. And so to that question of how do you ensure they're achieving the learning outcomes of the course because that's essential. I do kind of rely on both the fact that the final exam is cumulative and they have enough other things that even if they don't add on a slightly more narrow assessment of their learning, the other things like the readings quizzes, like the midterm and final, they're really getting to the overall learning outcomes. So I see variety in who's doing the best. And I also see, this is the part that actually really surprised me, is that a particular pattern does not predict what will happen for an individual. So in general, the scores on my exams tend to be a bit lower than the scores on the assignments, but not dramatically so like three to five-ish percent might be kind of the difference there. But I've seen particular students score literally 20% better on an exam than they did on their assignment and 20% worse on an exam than they did on their assignment. Like individually, like the standard deviation of it is large, the individual variation is large. So I look, there's noise, but no clear pattern that makes me think, oh my gosh, like anybody who's choosing this, it almost needs to come with a warning, nothing like that. And I don't plan to give them the, sort of what was akin to one of the examples I gave of, hey, your score will be calculated according to one of these things, whichever ends up favoring you numerically. Well, that's great. And that gives you further reason to believe that you're not unfortunately or arbitrarily disadvantageing a particular set of students for choosing a particular thing because, oh no, it turns out all the warning is only in this part of the course that you ignored it, seeing that kind of distribution and that lack of consistent trend is really encouraging. Yeah, it really is. And it's been really interesting to see because I speak to it to students in the sense of consider sort of what your own priorities are. Students often say what they feel they'll do best in, of course that's going to be a consideration that I frame it in terms of how would you like to challenge yourself this term? Like what would you like to do in terms of growing a skill, having a different experience, that it's not just how do you think you'll end up with the highest score? Again, because to me like the depth of the learning that can occur, if you're kind of pushing yourself like the hot topics discussions that I mentioned very briefly, they're a small group discussion, maximum seven students, so I aim for six, plus me having a pretty animated lively discussion about a hot topic in the field. That could be like something like, oh my gosh, can I like go in and do that? And yet students like love it and benefit it from so much. I've actually done a separate analysis to see among the hot topics discussion participants, how do they do on particular things? Because I think that direct involvement, especially in the context of a very large class can have benefits, but really I think that's, it's more than just the logistics of the app is really important for streamlining and making it feasible for instructors to do this. But I think it's the real framing of what are we trying to accomplish and how do we introduce it to students so that it's not just almost like a gamification of your learning in a negative way, there can be very good gamification of learning, but where students feel like, wow, like I'm empowered to actually choose, but if I feel like I don't know what to choose, I can trust that the instructor is saying, this is what will be best for most people, but that they have that latitude to make some choice. And I see so many like individual kind of examples of people that just, I don't know, they just grow into that, like people that maximize the value of particular assignments and like really invest or people that are increasing the value of final exams and boy, are they engaged in class. It's like I need to know. So yeah, so it's neat to see. Thank you. Yeah, thanks for the question. Any other questions that people might have? Oh, thanks, Judy. I see you put the link in for the paper. Super helpful. Go ahead, Jonathan. Hi, I was wondering if you could comment a bit more about the motivation for those students that actually choose like a custom great distribution. So sticking if I spend myself in their shoes, even if I knew the description of upcoming assignments, now as a student, you don't know what you don't know sort of thing. So it's hard to make sense of the content that you're going to learn and how hard or interesting an assignment will be and so on. So I was wondering if you know what motivates them to change things and mostly structural like how much time or frequency it's gonna take or is it somehow curiosity driven based on what they can gather from their learning outcomes? Yeah, yeah. And I think it's a little bit of all of those things for different people, which is also part of the beauty of the approach that it does accommodate so many personal learning journeys. One important thing to note in this course I've focused on here, though it's not the case for other courses I teach in which I've used this approach is this course is one that can be taken by students all across campus as an elective. So I literally, I think in this section have students from 55 different majors ranging from computer science to creative writing and engineering and everything in between. And so within that context there's so many diverse competing priorities that people have. And so among those that share because they're not required to there's a lot of things indicating like really things like work responsibilities and sometimes caregiving responsibilities. It's almost like I think sometimes if people feel they're opting out of something that they're more likely to say, like, sorry, I find it very interesting but I can't because of this. I've equally had students say something like I found it really intriguing is a word that comes up on some of the assessments which might be part of to that curiosity driven component where people are thinking like the idea of creating art to explore an issue do a deep dive, think about communication analyze the urgency of things related to the topics that we're talking about. People comment on it arising their curiosity or a number of students from the Faculty of Science who have commented that I actually really enjoy art or music but I've never had a chance to bring that into my academic life. So I think sometimes there's that sort of like huh, I'd like to do that, like that type of thing. So yeah, the combination I think of logistics what is the time they have available for kind of fleshing out their learning in this particular course is a constraint for some and on the flip side, how would they like to add to their experience in ways that maybe they just feel kind of compliments what they'd like to do at this point. Those are the things that kind of mostly come to mind. I also have a surprising number of students that go with the proposed grade distribution but tell me it's after a lot of thought that I decided to go with the proposed grade distribution. So like people do really kind of think it through and think, yeah, I guess that is actually what I would like to have. And I think it's funny because sometimes even though they don't need to be in touch people are be like, I know you said that and stuff but I just wanted to like confirm and those types of things. One thing highlight is a benefit of the app that's also something that I do in Canvas like up to this point the app hasn't been available is making sure students have a record of their choice so that they can see it in Canvas and refer back they know exactly how much each thing is valued. And one thing that I do in my current process is as I mentioned, I manage it by email. And so I do reply to students like confirming got it it's been recorded your final grade will be calculated like this. And with a little message of like wishing you all the best for the term ahead like something along those lines because part of my own belief is that anything we can do to shorten the distance between an instructor and the student in a large class like there's just under 250 students often taking this course. I also think that that's kind of like that little bit of connection motivation investment in their learning and those types of things. But the key thing would be to make sure they have the record which the app would be providing because they can refer to that in Canvas on their own. Makes sense, thanks. Thanks for your question. Thanks Jonathan. I actually just looked at the time and I saw it's four o'clock. So how about this? We'll wrap up so that we are precisely within the three to four timeframe of the session with tremendous thanks to all of you for joining us. Hope this has been interesting and useful for you in some way. Both Gabriel and I are certainly available for any outside of the session conversations you'd like to have and I'm able, I'm not sure Gabriel if you can as well, hang back now if anybody has any questions that they'd like to discuss. But we'll wrap it up there with best wishes for everybody. I hope that you're all doing well and just sending you all the very warmest of wishes as we navigate these challenging times. So thanks everyone. Thank you. Thank you everyone.