 additional territory shared between the Haudenosaunee confederacy and the Anishinaabe nations. And this territory is covered by the Upper Canada treaties and is within the lands protected by this dish with one spoon agreement. And it's directly adjacent to Haldeman Treaty Territory. So we acknowledge this because we acknowledge we have a debt to those who were here before us and we recognize our responsibility as guests to respect and honor the intimate relationship indigenous peoples have to this land. So I'm here to talk about, I was really, I loved Cyan's keynote yesterday about her travels to space and I was definitely one of the ones that put my hand up, I want to go to space. And I thought, oh, it's kind of, it's nicely tied to this project that I've been working on with a group of colleagues from other institutions in Ontario. And it is on beyond the exam and we kind of think of it as the final frontier when we're working especially in teaching and learning spaces and helping faculty adopt open pedagogies and alternative assessments that assessments really are something that often, often people even use open textbook and open materials, they still do a traditional exam and we'd like to change that a bit. So we had to, we decided to come up with this project and I'll talk a bit more about it now. So I'd like everyone to go back in time and maybe you have to go back in time a little bit more further back than others but to your own educational experience. So just think about that for a minute. What assessment did you learn the most from? So I'll give you a minute to think about it. Was it perhaps an exam? Was it an experiential opportunity? Was it a presentation that you had to prepare for a class? Was it an essay or paper that you wrote? Was it a project? Was it a portfolio or maybe it was other? So think about it for a minute and when you think about your experience of what you learned the most from an assessment, I'm going to ask for a, instead of a mentee or fancy schmancy way of getting your responses, just to raise your hand. So did anyone learn the most from writing an exam? No hands, okay. And that's probably the most frequent form of assessment. Did anyone learn the most from an experiential opportunity that was assessed? I see a few hands, a lot of people and maybe, I don't know what field you're in, but in health sciences and clinical placements might relate to that. Did anyone learn the most from doing a presentation in front of a group of peers? Yeah, that's me too. I find I learned the most from preparing. There's a lot of people in the room felt that. What about preparing a portfolio? There's a few hands, yeah. Yeah, and a lot of programs don't actually include that so maybe that's why we didn't all have that experience, but I think it's a great way to assess your learning, especially through the trajectory of a program. What about a large project? I wondered if that would be the most and I think it is, so that's a wonderful way of project-based learning of course is a great way to assess. And then other, maybe there's a category, any others that I didn't have on this slide that you maybe learned more from? I was hoping there'd be someone because then I'd ask you, there is, just keep that in your mind because at the end when the questions are, I'll ask you to share it. So as you can see, not many people learn very much from exams and that is the most traditional form of assessment, well, from my institution experience at least. So how we evolved this project was with the pandemic, we at McMaster and the institutions that I collaborated with, we all were faced with the realization or not realization or the unfortunate request that we had to obtain proctoring software, which many of us were opposed to and we know that there are many issues with proctoring software and some of them, but the basic issues are with technical issues, so it's not really, it doesn't really actually work, it's easily circumvented. Equity issues, more importantly, when students need to have access to robust internet, they need to be in a calm place, they are able to do an exam that's free of distractions, they need to have a robust computer and they need to also think about their own health and maybe they have an accommodation or a disability and the proctoring software just maybe gives an inappropriate red flag and then also having recognition of time and family responsibilities and particularly during a pandemic when everyone is already overwhelmed and having to do a proctored exam, we had discussions with many students who were just, it was really heart wrenching to hear of the experiences that they had with being flagged for inappropriately looking off in the distance like you do or maybe that was the color of their skin actually raised a red flag, which is really unfortunate and I think something that we should work really hard to not use proctoring software in education. So we did have to, we did get proctoring software anyways because some of the professional accreditation required an exam and so we couldn't tell the Canadian Medical Association for example that we can't have an exam, we're against it. So we had to use it in certain circumstances but when we talked to faculty we found that the biggest barrier to looking at alternative assessment was they just didn't have the support in looking in their courses and developing assessments so they needed instructions on how to integrate them into the class, they needed rubrics, they needed examples of the assessment that was being completed by other students and often we need, they also needed assistance with the technology if there was some technology integration so and we all, we realized that faculty were also similarly overwhelmed so they didn't have just learning to teach online for the first time was enough so they really needed that support with these type of inclusions, the instructions of rubrics, examples, etc. So we were really fortunate in Ontario to have eCampus Ontario which many of you may be familiar with if you know David Porter or Lena Patterson who often come to the OE Global, Lena is the chair, they had a virtual learning strategy funding that was available two years ago and we applied, they had a category digital fluency for educators, we had a collaboration, so McMaster University where I'm from, we worked with Brock University which is in near Niagara Falls Ontario and then College Boreal which is in northern, it's a Francophone institution in northern Ontario. So the three institutions applied, we received some funding and we were able to get the project going. We used the funding largely to pay students and conduct French translation. So we formed our development team, there was multiple people from each institution, we had faculty representation, teaching and learning staff and students as well. We connected either through real-time virtual meetings or through Microsoft teams and have developed the project. So when we first decided the outline of the project we knew that we had to have, we really wanted to build it around the exemplars and the kind of documents that instructors could, really requested that they needed for their courses and we used a lot of our student discussions to come up with the type of assignments that the students found most valuable. So we basically outlined the projects, having an introduction, some bank of exemplars and we built this in press books and we also have a WordPress landing page and again we did translate it into French. So that was also part of the budget. So let's take a look at it. So there's the website beyondtheexam.ca. This is our landing page, hopefully it goes up okay. There we are. So here we are landing in our beyond the exam and we have a little bit of information around what the toolkit is. There's the English version and the French version. So you can access either. But an important part that I wanted to point out is do you have an alternative online assessment to share? So there's a little submission form here. So if anyone has an interesting assignment that they'd like to add to our bank of exemplars, we have about, oh I should have counted, we have about 24 or so across disciplines, across like from an introductory level class to a more advanced group projects, individual projects, formative and summative assessments. But we really want to build this out to be a big robust alternative assessment bank. Let's go up back up and take a look. I'll just let me know how I'm doing for time too. Am I okay? I'm looking at my facility. Okay, thank you so much. All right, so let's go back here and I'll show you an example of one of the exemplars and I'll point out a couple of things. So we did a little bit of custom styling to pressbooks because we wanted it to look a little bit snazier. And here we are over contents. So the introduction is here. Sorry, I'm not used to these windows. So we had a student perspective just talking about how they wanted alternatives to conventional exams. It's a great opinion piece written by our students. We had the instructor perspective, which is a wonderful faculty member from Brock University. And I am totally in alignment with her style of designing a course is after week eight consolidates. So after week eight, she does not introduce any more material. That's when you assess all of the content that you introduced in the first eight weeks of a traditional 13 week course. We also applied Blooms taxonomy of everyone anyone's familiar with that taxonomy of learning and how you scaffold to deeper learning. So we did tag each of the assignment exemplars with the Blooms taxonomy levels looking for those, you know, higher levels of analyzing creating. So we talked about that in case we just gave an overview of Bloom. So in case people weren't familiar with that. But the bulk of the book or resource is in this, let's see if I was right 20, I said 24. Okay, I'll take a look at this one. This is one of since most of us mentioned project as being one of the most pivotal assignment examples in your educational history. So here's an example of a multimodal culminating project. So each example we use the same format. So we have a description of what the assignment is. We have guidelines. There's the guidelines on this particular assessment. So we could do a multimodal narrative. You could do a multimodal presentation. You could do a media artifact. We have proposal documents that can just be copied over and used for the instructor. There's a metadata document for for students feedback in Cacheek. So how the project is assessed. There's the evaluation criteria there. So we could use that for development of a rubric. And then we also included technology used for each of the examples in case they did use a particular technology needed a little bit of tips for students on how to use that. We also include facilitation tips for the instructor. So just how the project or how the assignment had gone historically. What what the student feedback was like. Maybe things that you might want to consider if you're considering implementing into a class. And then we also included student examples. So that was the basic format for each of the assignment examples. And that's so and so our our hope and dream for this is that we have more assignments added more exemplars from each of you maybe. But we also have other plans. We want to keep this project sustained and used. We originally intended to include a workshop design. So a whole part to the press book on how you might if you are in a teaching learning profession facilitate a workshop on the on alternative assessments with you know slides and activities and and and stuff like that. But also have a self paced opportunity for people who just want to go through and consider it on their own. So that is underway right now. It's not quite ready. We're the English version is complete but the French version is currently being translated. And we also realize that you know we have a small bank right now that we started with. But we want we hope that this will grow to be hundreds if not thousands. And if press books doesn't actually isn't actually the best way to host these that might be confusing. But we have to figure on how to categorize those assessments by individual group formative summative maybe by disciplinary area. So that's that's our next steps in this project beyond the exam. And we are I just want to do another plug for more exemplar. So please consider any examples that you'd like even if you're thinking Joanne I have this idea that I really like to talk about. I don't have all the instructions I can interview you and get all the instructions and do all the work for you. So we really would like to see that grow. So thank you so much. I just wanted to thank the McMaster Brock and Boreal Development teams and all of you for your attention on this. If you do want to get in touch with me either here I'm wandering around the conference but you can also email me or find me on Twitter. So thank you very much. Well there's a mic up here yeah. Thank you very much Joanne for your presentation. It's very interesting but my question is not for you. It's for the two people on the on the room here who said that they have all the kind of exam or something like that. I would be interested to hear about it. Oh the other the other projects. Yes. Yeah. The kind of project just to know so sorry about that. Yeah okay I want to bring the microphone down because it was you were sorry microphone I can actually bring one. Oh no there's one wireless one here. Oh here it comes. So anyone who I think it was anyone who said that they had something. Okay you're pointing. There yeah and you yeah. So yeah thank you so much for this presentation and I think the first the introduction with how do we learn more with which exam it's very very relevant because we have loads of exam and QCM and it's not the way we learn and the project based exams is really really impressive and I'm using it but when I think about how do I learn the more the most it's without any evaluation. It's my free will and my interest and it's very related to the project I'm working with. So it's I think the project base is very interesting because it's leading the people to what they will do in the future as well. It's preparing them to learn by themselves to achieve big project. That's a really good point and that's actually when we're talking about the project of course we were talking about ungrading and how we maybe we how do we capture those it really is an experiential learning when you're working on a project and learning the most and how do we capture those. So definitely how to be more creative and more personalized I guess into people how the way that people learn but I was wondering your your yes down here you had the um you put your hand up for an other assessment that was the most valuable for you and what was that. Well first of all thank you very much for your presentation Juan and maybe the first thing that the question that you have asked from what which assessment have you learned much brings me to a reflection it's not yet mature in my head but I think that the future would be like learning activities and assessment activities will convert so they wouldn't be two different things yeah because actually we assess to make sure that they have learned and if the mechanism is the same to assess that learning has occurred and then we don't need any other assessment mechanism and actually that driven me has driven me to three assessment methods that I had in my classes and for which I have an excellent feedback from the students and really serious students that I know they want to learn they don't want to grade. The first thing was simulations I'm an information systems professor so I teach ERP technologies and things of the sort and if you just tell them what is enterprise integration they wouldn't they wouldn't understand unless they were different hats so the project is just simulations about mega softwares where they play roles of production managers the warehouse whatever procurement specialist and they do the whole thing and actually maybe in a day or two they understand the whole curriculum of one year where you had to explain different modules different rules different privileges and stuff like that. So simulations was one the second thing I used and that was also very well appreciated is evaluation through content development there is a famous saying of Albert Einstein that says that I cannot really don't take me by the word but the meaning was if you cannot explain something so probably you don't understand it from the very beginning. So I brought my students to produce pedagogical videos about technology and technology concepts and technology usages and I was really they proved me that they understand everything first second I think that they have even presented bits and pieces that I couldn't present myself in this way. Yeah so this was great. The third was debates driven by learning objectives of a certain course so I had this was very difficult to scale because debates were different topics and you had to they had to defend against different questions and this had been you should have been leveled those are great again thank you. No no I'm answering the question that was asked. Okay so they were leveled to level them because it's not the question addressed to everyone or something is the same exercise but this also did great because they could find arguments and statistics and different things from the curriculum. That's great thank you so much for that and I know that I'm going to be contacting you for examples. You know I'm a Canada graduate and I'm very much voice to Canada. Oh great. Congratulations. Yeah thank you so yeah simulation teaching a lesson as kind of the second and then the debate are wonderful examples yes thank you. Thank you very much for this presentation and this idea it's great. I wanted to know about feedback so what do you mean by feedback is it I don't know is the results of the exercises or yeah a sort of satisfaction right from the students or both actually yeah it was we had feedback from from instructors who had implemented the assignment example you know how they found it what tips they would give another instructor so that kind of feedback and also the feedback was as what they heard from students. Yeah usually it's just data that we are already have collected through a course evaluation so that is shared back anonymously of course to us yeah great thank you. Hello I'd like to ask you a last question maybe yes so thank you very much for your very nice presentation my question is regarding actually actually my colleagues asked me first to ask a question regarding the licenses for for sharing yes and the second one my question is regarding the standards or the norms that you may be used for for the design of the assessment whether you are comfortable to IMS QTI question and test interoperability standard which makes maybe the operability of sharing and exchange exchanging these resources among platforms. Yeah thank you. Great thanks I think all of the assessment examples anything contained in the book is CC by NCSA so freely used and adapt to and yeah so I'm not sure I understood your second question I'm sorry yeah the format if it's adaptable definitely yeah and we're even finding that the technology used we didn't really apply in some instances it's like a learning management system or a word you know so yeah we definitely see that we see some adaptability that we might want to add different categories yeah thank you. I think I have just one last question. Oh sure. Just a quick one where do you see this being you know transformed or used in the next two years? Well that's where we want more examples to come in and to become like the source that people I know there's like we're so lucky we have so much so even when we first were helping faculty with that alternative online assessments there's so much out there available so we're just looking to hire more student development teams and getting asking instructors if we can use and build to our bank of exemplars so we have a lot and have them categorized so people can easily find them so communicating and disseminating and and hearing about how people with value that people find with the resource and and growing it from there yeah. Thank you very much. Thank you. Thank you everyone.