 So good morning. I think we'll go ahead and get under the weight. I wanted to thank everybody for being here today. I feel like I've been saying thanks for coming to our workshops. It's a very busy time of year for the last two years at any given time of the year. But this is actually the first week of classes on campus and really appreciate people taking time to be there. My name is Will Engel. I'm a strategist for open education initiatives at the CTLT and I'm joined by my colleagues and Trish and Pauline. I'll let you introduce yourselves. I'll go first because I muted myself faster. My name is Trish. I work for the subject for teaching and learning, teaching learning technology, and as well as part of the Institute for the Scholarship of Teaching and Learning in the capacity of an evaluation and research consultant. Thank you, Trish and Will. And yes, my name is Pauline and I'm also part of the ISODL Institute, the Scholarship of Teaching and Learning at the CTLT. And I work as a SODL facilitator and evaluation consultant as well. So I'm really excited to have Trish and Pauline here talking about evaluating open education resources and open education projects. OER is making a huge impact at UBC in the 2020-2021 academic year. So last year we estimated there were roughly 19,000 UBC students who took part in over 60 courses or course sections that had replaced paid textbooks and were using open or freely available resources in their place. The goals of instructors and faculty who are implementing OER projects are really grounded in the idea of improving learning and enhancing the student experience. So I'm really excited to hear Trish and Pauline talk about how do we know if we're implementing OER into our courses or open educational practices into our courses. How do we know if those projects are actually meeting the goals that we set for them? So before we begin, I would like to acknowledge that UBC, which is hosting this session, is located on the traditional ancestral and unceded territory of the Muslim people. And as many of us are still working virtually, I'd like to acknowledge that here in the lower BC mainland we're often on the unceded territories of many different Coast Salish people and outside of this area. We may be on the territory of many different indigenous peoples. And when I acknowledge that I'm on the territory of the Muslim people, it's rooted in the understanding that I, as a member of UBC, am privileged to be working in a territory that is not my own. And then because this workshop is focused on open education, and because my role is specifically focused on open education, I also do like to acknowledge that much of open ed is grounded in Western notions of copyright law and ownership. And these notions can be in tension with indigenous and traditional ways of knowing. We're not going to be talking about those tensions in this session, but we do have an upcoming workshop in March that we'll be talking specifically about those tensions, so I'll drop a link into the chat in a sec. But I do just like to acknowledge that upfront that those tensions do exist. And with that, I'm really excited to turn it over to Tricia and Paulina to talk about evaluating OER projects. So thanks so much, Will. So I'll kick things off. As we get started, just a reminder that the session is being recorded, and we're happy to share the slides with you. After the fact, do feel free to share any questions in the chat. Paulina and I will be interchangeably monitoring them and hopefully be able to answer your questions along the way, but we should also have some time for you at the end. So I have some objectives to outline the session today. So hopefully as a result of this session, you'll be able to understand how evaluation can increase your project success, as well as have some clear evaluation objectives. So really honing in on what the intended outcomes and how to measure what that success looks like for your project, as well as provide some ideas on potential methods that will help you reach these goals. Just a reminder, this is really the beginning of a conversation. You might be at different stages in evaluation, but this is just a very quick one hour session. So we're here to kind of provide some tips and tricks and overview, but we're also here to, if you need support or further guidance in evaluation at any stage, do please feel free to reach out to us. And an outline of the session overall will be looking at some big picture of evaluation, what evaluation is and what the different components of it are, as well as doing some activities with you, looking at defining your intended outcomes and impacts, as well as discussing what your evaluation question might be, whether you have it solidified or not. And then we'll talk about some different evaluation measures and approaches, and then some overall considerations that you should think about when embarking in evaluation, and then we should have some time at the end for Q&A. Okay, so you're all here because you have some evaluation goal in mind, presumably. So hopefully this should come as too much of a surprise to you, but when we think about evaluation and kind of what the value of it is, it's really about making sure that your project outcomes are met and that your plan is working as intended. So, you know, you've set out to create some type of OER, and it's not just a matter of creating it. That's well and good in one piece of it. But is it working? What about it is working? Is it, you know, helping students learn better? Is it saving students money? Is it making things easier for the teaching team? Is it allowing you to incorporate new elements that you wouldn't previously be able to? An evaluation can help you answer those questions. So, evaluation in the sense helps you, as I mentioned, determine if your objectives or goals are being met. Also, improving project design. We think about evaluation often as an iterative process. So you might go through and ask some specific targeted questions in one way, and by asking those questions, you'll be able to learn something perhaps that you didn't know before, which could lead you to make some changes to the design or to the resource itself, as well as just making those informed decisions. For those of you that are here as OER fund holders, just a reminder that project evaluation is a required component when you complete your final report for the project. Okay, so we look at evaluation, as I said, as this iterative process and these different stages include defining your evaluation or research questions, looking at different measures. So what is it that you're hoping to evaluate? So, for example, something like, you know, does this open resource impact student motivation or student learning? Choosing an evaluation approach from there. So how will you know if that's actually happening? So thinking about things like doing surveys or focus groups interviews, then preparing and piloting your evaluation measure and then collecting and analyzing that data. So just to give us a sense of kind of where people are at, and then we can maybe spend a little bit more time on each of these areas depending. I'm just going to share a poll with you and just ask that you select which stage of evaluation you're at. And it's okay if you're really, you know, really at the start of defining questions, you haven't even started to define the questions, you're just thinking about defining your questions, that's okay. It's possible that you've already gone through and collected some data and now you're kind of trying to figure out how it applies to you. So just give a moment for people. And it's possible some of you aren't even doing an open project date, you're really just kind of starting to think about this process that that's totally okay as well. So it looks like most of you are in the defining questions stage. So just more than about half of you, but some of you are a little bit further along already doing some preparing and piloting or in one case having already collected and analyzing some data. So that's helpful for us to know it looks like more at the front end. Perhaps some of that piece will be useful and we do have some content on that. So as I said, this first stage that we consider is really in defining that question and in developing a question we tend to break it down into two pieces. So one piece being the practice what it is that you're hoping to evaluate and then what the intended outcome is so what impact you hope that practice to have. And so I'm going to work through some examples as we as we talk, but again, we're really in this context, we're looking at practice as an open access resource. So this could be something else in a different teaching context, but for the purpose of today's session, we're really focusing on that as the practice or sort of output. What you're hoping to evaluate. And we recognize that that resource could come in many different forms. So a textbook website, a teaching tool, et cetera, et cetera. And then when thinking about outcome, we think about that as what impact you want the practice to have. So just some examples here and we'll work through some more in a moment. Are things like performance, a motivation or attitude to change increased awareness to different diversity issues. And the list goes on and on. So as I said, I'm just going to work through an example here with you. So in this example, I, if I was will had thankfully given me some open. I'm looking at developing an open self study quiz database for all sections of psych 217, which is a wide research about this course that loads of students take. So I want to create this database that can be used. So that's the practice of what it, excuse me, what it is that I'm the resource that I'm creating. And then thinking about the intended outcome in my example, my goal is to increase student learning and knowledge as a result of these resources. And it's really important to kind of lay out these two pieces clearly before you build your evaluation question. And we'll get to that stage in a moment. So here with activity one. I'm going to share. So we have some worksheets that we've put together and we're going to work through a few of the activities together. Let me just share this chat. And so I invite you to make a copy of these worksheets and keep them and use them both today in this session as well as as you develop your evaluation plan. But what I'd like you to do is open the worksheets and then under activity one, which provides a similar listing like this, select which outcomes you hope to achieve from your OER. So think about, you know, what it is, what are the goals of your project and it's possible there's something else on here that's not or there's something else that's not on here that you're hoping to evaluate as an outcome that you're hoping to look at. Please feel free to share that in the chat. This is a list that we've put together based on other teaching and learning projects that we've supported at UBC, but it's certainly not the be all and also we're always happy to learn about different areas that people are investigating. And then what I invite you to do is just annotate on this slide itself and let us know. Once you've kind of thought about what your intended outcome is, let us know what it is. So I'll give people just a couple minutes to do that. And during that time please feel free if you have questions to type them in the chat or just pop unmute yourself and let us know. Did you want us to annotate this slide? Yeah, that would be great. Just to kind of get a sense of where people are at. It's sort of a nice, we're a pretty small group so it's nice to sort of see what areas people are looking at and get a sense of where the work is. I'm seeing lots of check marks coming up. This is great to see. It seems like a lot of folks are interested in this area of exposing students to different ways of learning. And that's fantastic. I see a comment in the chat here. Another one, students awareness slash exposure to diverse voices. Awesome. And I'm seeing that you're having some trouble putting your answers in the document. If you mean the link that I shared, you'll have to make a copy of it. So we actually, those worksheets are for you to use for developing your own evaluation plan. So we don't want everyone writing their answers on a shared document there. So the Google Doc, you should be able to select to save it either as a Word document or save it as a Google Doc yourself. Save a copy, sorry, I should say. Great. Okay. Just in the interest of time, I'm going to move on from here. Thanks everyone. It's always interesting to kind of hear and see where people are thinking about their projects and what they have in mind. And I think helpful to kind of, yeah, see what other areas that you might be interested in. Okay. So building on that, though that you've set up your ideas of what your practice are and your intended outcomes. This is a good way to sort of build into what your evaluation question or questions are. So from the example that I mentioned earlier, looking at developing open self-study, an open self-study quiz database with the goal of increasing student learning and knowledge. I could build an evaluation question, which is how do the self-study quizzes increase knowledge of core concepts? And you'll notice about this evaluation question and it's one small change that I recommend making when you're building an evaluation question is the use of the word how. So this could be easily phrased as do the self-study quizzes increase student knowledge of core concepts. And that gives you a simple yes, no question. But as part of evaluation is really moving beyond that yes, no, is it working? Is it not working? But why is it working? Why is it not working? What's working well? What's not working well? So thinking about that framing of the question also sets you up when we get to the stage of evaluation methods, building those questions that you would actually ask the students or the teaching team or whoever you're interested in targeting. So for this next activity, like you two again, go to your worksheet and in activity two on the next page build in some evaluation questions. So again, even if you're not really at the stage of being ready to get there, thinking about what you indicated on the previous slide as what the outcomes were and the practice or the resource you're developing. What type of questions might you ask? So just ask that you spend maybe just two minutes thinking about what your evaluation question might be and then making some notes for yourself, either on your worksheet or on a piece of paper. And then I also invite you to share them in the chat and pulling and I will kind of scroll through them and provide some feedback on that framing or we can provide some clarification. If you're like, does this sound like I'm down the right path or is this too broad or too specific? So I'll just sit quietly for maybe three minutes while people do that and then hopefully we can have some discussion around the questions people share. Sorry, Linda, it's your question here. How many outcomes are too many outcomes? That's a great question. I think it's fine to sort of dream big and think about all the things that you're interested in asking or that would be informative to your project. So I think it's fine to start off and just take off as many of those boxes as you're interested in. I think from there, I would say focusing on two or three for your first round of evaluation just because it's not so simple. Say you're interested in student motivation. It's rare that you would ask just one single question to students about motivation. So you kind of have to think about for every outcome that you're interested in that you'll have a couple questions that you'll want to ask about that piece. And so both in terms of for yourself in terms of analyzing and processing the data and students or the teaching team, whoever will be answering those questions for their sanity. You kind of want to focus on just a few areas to start. And I think that's a good way to start kind of for that first iteration of evaluation. In projects I've supported oftentimes, we'll start off with a few questions and we'll get really clear data on one or two of them. Students do really feel like the resource is valuable. They are feeling like they're learning from it. And that will help generate a new question that's maybe tied to another outcome. Or I can kind of move on from that and say, okay, it seems like this resource is really helping students stay motivated or understand the goals of a specific assignment or something like that. Okay, I can leave that one aside and then build in like, okay, well, but what about this other piece? So I think starting off with maybe two or three or even just one is totally fine and a good way to start and then focusing on those as the forefront. And yeah, that can take a bit of time to kind of narrow down. Okay, well, which ones are most important and most valuable? And it depends also on kind of the depth of the outcome that you're interested in. So it'd be easy to say something like looking at student learning, you could just say, well, their grades are improving. That's enough. That's all I'm interested in. But many of us I think don't want to, don't want to, don't like to think about learning as just a single grade. And so there's a little bit more richness to it. So it also depends on kind of the depth of the outcome as well. Hopefully that answered your question. So if anyone else has any questions or they'd like to share their research questions that they, or evaluation questions that they've developed, please feel free to post them in the chat. We have a few minutes that we'd be happy to spend kind of helping people finesse their questions or figure out if they're on the right track. I'm also mindful that it's fairly and maybe people aren't ready yet to share their evaluation questions and need a little bit more time to process. This certainly isn't meant to be a, you know, at the end of this session, you'll have finalized everything. So please don't feel like we're trying to rush you through this process at all. We're happy to provide feedback. Also, as I said earlier, you know, take some time, think about this, put it, put it aside. You're always welcome to come back to me and Paulina to help get some clarification or guidance. I'm seeing a question here. And other questions. Yes, no problem. So how does the resource address the gap in existing resources? Yeah, that's a great other sort of outcome is sort of, and that's I think a much bigger research question in terms of how does it compare to what what is already there? What's already being used? Yeah, this takes it down a little bit of a different route than sort of what we're focusing on today, which looks at more gathering people data. I mean, that's certainly something that you could think about doing with this question, but kind of looking more at the literature or other experiences and saying, you know, surveying the field. And yeah, as Paulina added here, what is it adding that that isn't already in existence? Awesome. Thank you for sharing that. Okay, I'm going to move on just being mindful of how we have a lot of other content to share with folks. Okay, so once you've kind of narrowed in your question and feel like you've got a few that you're ready to tackle. The next stages are identifying measures and evaluate choosing an evaluation approach. So looking at these. So again, I'm just building on that example that I mentioned earlier. So how do the self study quizzes increase students knowledge of core concepts. Two ways that I might think about measuring that are looking at student performance on knowledge tests, as well as looking at student confidence. And I might think about in this example as evaluating those via quizzes. So looking at performance. The grade changes based on quiz performance and look at student confidence either by using surveys or focus groups. So the measure is really an indicator what will help you find the answers or signal that a change of some kind is occurring. And then your evaluation method is kind of how are you going to do it? Where are you going to find the evidence? What data sources or approach are you going to use? So we don't have time today to work through that and also recognizing that people are still earlier on in that sort of developing their research questions. This is something to do once you have that little more solidified. So we won't spend time on activity three today. But I do invite you to go back to those worksheets once you're here and use them to kind of help you frame to think about the measures that you could be using. There are some examples there of different ones for different target areas and different thinking about the evaluation method. So they hand it over to Paulina to talk about some common evaluation methods. Thank you so much Trish. Yeah, so in the poll that we did earlier actually nobody replied that they were at the stage of thinking about evaluation approaches. So I'm not sure if that's because some of you have already completed that stage or if you're still all very much in that very first kind of thinking about your research question or what your project seeks to do. So I won't spend a lot of time on these slides because I definitely want to make sure that we have enough time for your questions and I see some more questions kind of popping up in the chat. So obviously if you have questions while I'm moving through the slides, please do feel free to unmute your mic and then or just pop your questions into the chat. So let me see if I can oh Trish, I can't advance my slide. Oh, there we go. It's just a bit delayed. There we go. So, so common evaluation methods and so I'll spend a bit of time talking about the most common evaluation methods but this does not mean that there are not other ways that you can evaluate your project. So interviews, focus groups and surveys are probably the most common methods that we see in OER projects, TLF and also the subtle seed projects that we support. But there are also other ways that you can do this so you can do it through observations so looking at how students interact when they're doing certain activities in the classroom. You can look at assignments, so quizzes, tests so that's kind of more if you're looking at evaluating like how well students are learning or are their grades improving and that sort of thing. So you can also look at journal reflections. I worked with a project once that looked at student written responses based on certain activities that they did. But eventually the evaluation method that you choose really really needs to tie back to your question so if you're thinking about for example looking at student engagement or how the OER resource help them understand diversity better. There are many types of things you might want to think about in terms of interviews or focus groups because you can really elicit rich data, you can look at student experiences, lived experiences in different ways than for example a survey can capture. So it's not to say that there is one approach that's better than the other, but the question will dictate which method that you use so it's really important to kind of think about those two things together. And it's also really important to think about for either method any method that you choose to think about how that information will be used. How, how are you going to make sure that students have access to that data if they if they want to look at what was done with it, making sure how you're going to ensure student anonymity. So all those things need to be considered no matter which method that you choose so that's something to to really think about before you move on so. So, interviews and focus groups, as I mentioned, these are pretty common, usually in the social sciences, I've seen them used a lot or in the arts kind of fields, where instructors are really interested in kind of that that greater picture of student experience. They give you a detailed understanding, they, you know, a lot of stories and narratives come out of from talking to students. You'll only get a small subset of students to participate in interviews and focus groups, interviews will definitely take a little bit longer to do because usually they're one on one. And oftentimes we need to find someone else that who is not the instructor to conduct those interviews because of that power dynamic and so you might want to work maybe with a GRA or a grad student who might be willing to conduct some of those interviews a little bit more of that time intensity that comes along with interviews and focus groups. The other the other thing to consider, you know, as great as these approaches are, they do not often generate high attendance so oftentimes focus groups are not well attended. There was an instructor I want to mention who in this who's in this presentation today, and we worked really closely on with focus groups, a couple years ago on a project. And we had very, very few students attend the focus groups, but the information that came out of them was really, really enlightening and really helped us kind of think about how this instructor could move on with the course in the next in the next term. So, even if you have small numbers, don't be discouraged, you can still get really interesting data from that. So those are just a couple of things about interviews and focus groups. I'm sure that you have a protocol is really important so having maybe like an interview guide or something where you can follow along as you go with the interviews or focus groups, especially if you're not the one doing it so just making sure that those questions are being asked and that you're asking them equally among all students that you have the data that you need. If you're not familiar with conducting focus groups or interviews. You'll learn more about how you can do that of course there's workshops that you can attend to learn more about it but as I said it's always better to have someone else do them for you. And if they're not familiar then it's always good for them to have a bit of training as well because sometimes, you know students will share things that you'll need to just have a little bit of experience and navigating some student comments, especially that might be sensitive around like course material and instructor and that sort of thing. And always making sure that you record the session with permission because you will not remember everything or the grad student that's working with you won't remember what the students said, even if they're taking excellent notes so recording is usually recommended for that. Are there any questions about interviews or focus groups or anything that you've been really wanting to ask about some of these methods. And we'll talk about ethics a little bit later on if that comes up but I'll just keep moving. Okay, surveys surveys are super super common, mainly because they're very very easy to administer they're really easy to create if you use something like Qualtrics. Even if you have no experience designing surveys Qualtrics is a really neat tool that helps you kind of through different questions and it gives you tips on ways that you can restructure some questions if they're a little bit problematic or if they just don't make sense in that in particular Yes, and we have a workshop coming up at the end of March on survey design so if you're like oh I think I would really love to use surveys. Then please join us for that and Trish is the survey expert so she will teach you a lot about survey design, but they're really great if you have a large number of students. So if you're evaluating something like you know student engagement, and you, and you really want to have like all your students participate in that you're likely not going to get all the students to come to a focus group. So a survey might be a really great way of asking some of those questions about, you know, didn't your tool, you know, make the students feel excited did they learn did they want to engage with that tool did they did they feel that they could engage with it more than like a textbook or a traditional type of type of type of learning tool. So you'll definitely get a little bit more data from surveys, but the thing to consider about surveys is that there's not always a good like there's there's different times during the term when they're really good to send out to students. So you have to be really careful and mindful of, you know around exam times or end of term when students already have so much stuff going on, or maybe they're going back home for the summer. They're planning out the time the timing of the surveys of when you send them out and that will really determine how many number of students you get to participate in those. There's a question if you design survey and our focus groups questions together with former students. Nice to learn more about design. Yes, are you asking if you if you can do that Karen or if that's something that we would recommend both. Okay. I would say that's a that's a really great idea and that's something that's, I guess in line with the student as partners, kind of initiative and and trend that's that's really happening right now in the university so I would say yes to that. There's probably a little bit of planning involved in involving students so making sure that that there is some kind of protocol or ethics screening there so that they're in because they're basically involved as researchers in that in that project with you. And so yeah I would say sorry I'm just trying to look back at your comment here. Yes, and so piloting piloting with students is always great for a survey but also piloting maybe with with faculty, or any of your colleagues that might want to have a look at your survey just to make sure that it makes sense. Sometimes piloting with students again we have to consider the time that students might be investing in this. So I would for piloting a survey I would say I would recommend it with with your colleagues if that's possible. Trish I don't know if you have any, anything to add to that I would just say like really being mindful of students time. And I would just grab maybe like three or four colleagues who could look at it and make sure that the questions that you're asking. Obviously they make sense to you if they make sense to to other people, but I yeah I guess you could pilot it with students as well if they have time and they're willing but I guess I would just be really careful that that they're not overburdened and that they don't feel that they that they have to participate in something like that so coercion is a huge thing sometimes with projects when we when we ask them to get involved so it really just. Yeah depends on the specific situation Trish that's an important point so I just want to make sure if you have anything to add to that. So that I think I'm just looking at your your clarification here talking about piloting the final product so yeah certainly great to kind of do kind of a more targeted. Think of something like like a user experience evaluation where you kind of say like okay before we share this with potentially hundreds of students. Does it work out like kind of iron out any bugs I think that's maybe what you're asking about is kind of getting a sense. Before you hit the ground running and definitely we would recommend that so thinking about piloting, you know before you really dive into specific evaluation questions about you know how is it impacting or learning or motivation or any of those other questions just getting a sense of. You know, does it work. Are there any major kind of concerns that come up. The other thing with piloting or another way of thinking about piloting that Paulina was kind of, I think alluding to was also piloting your evaluation method materials so say you have. A list of survey questions that you're going to share or a list of interview a focus group questions, having someone read through them. Someone who's not going to participate just to make sure that you know there isn't any strange jargon or you know thinking about things like you know does it say. Does did you like this and it's sort of like a yes or no, how you can change those questions so getting someone to sort of read through it and pilot it in that sense is also really useful. Yeah, I hope that that kind of helped clarify. And if you don't mind I'll just jump back to Karen's comment that doing that with students who have taken the course before is a great way so so. Sometimes people might be evaluating sort of student attitudes towards the resources being used and and piloting sort of survey questions or helping design survey questions with students who have been in the course before. Can raise things that that might be blind spots so like maybe students get really frustrated that some of the topics aren't covered or or that they can't access those materials in the way they would like to access and you might not be aware of that. So it's often great to partner with students I think. Thank you will and Lindsay you had your hand up as well. Yeah, thanks. I was just wondering, and you touched on this a little bit, but you know surveys, you can get so much more data and they're a lot easier. So I guess I'm just wondering what what would make you lean towards choosing a focus group over a survey. For example, if you're measuring student knowledge and engagement. Like what I'm at that stage I'm like how which one am I going to choose. So just wondering what things that might sway you towards focus group over survey. I think also it's important to mention maybe I didn't make this clear that you can do both. You can do interviews and and focus groups and a survey. If that's where you want to go and oftentimes we actually encourage people to sort of use mixed method approach because you can get kind of like a wide range of different perspectives from a lot of people. If you do a survey but then you can also kind of narrow down on the more like nuances of this of the student experience through interviews or focus groups. I think if you're not sure it and you're sort of like I would I would like a lot of data I have a lot of students in my class for example, and I want to make sure that they all have a chance to, you know voice their opinions on this tool. So you can do survey, but then you also might have a couple of questions that you're really curious about that you can maybe only get through talking to students so then you might choose to invite, you know a small number of them to participate in a focus group and so I think that can make it for a really, really rich project so you don't have to choose one or the other and we often encourage people to do both. I hope that answers your question. That's great thanks. Okay, and just one thing about surveys is keeping them really short so I would say no more than 10 maybe max 15 questions on a survey, because the more questions that you have, then you really have to think about well maybe actually I am interested in really talking to students and learning more. So maybe a survey is not really the best method for you. So, and again, keeping in mind that after about five minutes of doing surveys students tend to tend to get really tired and so the quality of the responses might might not be as high so keeping in mind the the length of the survey. And then also, when you choose to administer that survey like what point in the semester do you think that they're not going to be super overburdened, you know during exams or during reading week when many of them might be away so things like that. And we can always provide support, you know with with questions like that if you're like not sure. If you're thinking about administering a survey we can always give you tips depending on your specific course so feel free to reach out to us about that as well. Okay, so preparing and piloting so we're kind of talking a little bit about piloting a little bit, and then data analysis and data collection. So I'll just touch very briefly on that but before that, creating an action plan so this is really important. And activity for, which is the last page of that worksheet that Trish shared with everyone has a really good sort of framework for how you can sort of set up the project in ways that you can make sure that everyone who's in on board with the project. Knows what they're doing and that you have specific timelines for things. If you're the only one doing the project it's a little bit easier but if you're working with like I said a grad student, or maybe another colleague, it's really important to have those things laid out really clearly. So who's doing what so who's doing focus groups or who needs to read, you know design a survey and by what time. And if you are going to be applying for breb, for example, then you need to sort of plan a little bit more in advance because sometimes applying for breb can take a little bit of time. And so, you know, thinking about what things you need. If you are thinking about very kind of like practical things like for a focus group you need to organize some food or some gift cards for students. So really thinking about all those things in advance can really help you in the end and save you lots of time. And then of course always thinking about how can you tell if a milestone has been met so always going back to that evaluation question how do we know that what we're doing is working. How do we know that what the kinds of questions we're asking are helping us to evaluate this this OER project. And so using that activity for I think can really help you when you get to that stage. So just to consider. This is something we see not just an OER projects but TLEF and subtle seed projects. This idea of, oh I want to really look at something that's interesting in my classroom. But just because it's interesting doesn't mean it's worth evaluating and so really thinking about, not just whether it's interesting or whether you're interested in the staying or you're really passionate about learning more about it, but whether it actually can contribute to something to the student learning experience. So it shouldn't just be something that you find interesting, but something that can really benefit the students in the end. And so oftentimes a lot of people might find that it can benefit students but really thinking about does this project benefit student learning. Does it benefit them, you know with their engagement. Does it benefit them in learning about diversity. So if you if you answer no to a lot of those questions and you still think your question is interesting, then it might be worth maybe talking to us about that and we can maybe help you redesign your question in ways that actually does benefit the students. And because they should be really the focus of these projects. So about you know I understand my project I understand my survey I understand these questions may not necessarily mean that your participants understand. And so always again going back and piloting. As Trish mentioned piloting something before you get going, or asking your colleagues for feedback on things that you think make sense to you but then, you know when you send them out to students they might be super confused and then your data might be not so great. And so always answering randomly perhaps. Also thinking about maybe ways of integrating the evaluation into your course. So sometimes it's common to have things integrated into a canvas course. So for example if you have an assignment in canvas you can use that as part of your evaluation so sometimes you don't need to make those extra questions going into a focus group or creating surveys. Think about how your course is laid out and think about whether the way that the students are moving through the course if you can use some of those things that they're already doing as part of your evaluation. And of course considering the time and the cost to administer evaluation. And then of course the big question that we always get that we probably will, we'll get at least one question about ethics today, which is ethics do we need brev approval and this is not a clear yes or no. In either, in either case it really depends on what you intend to do in your project. But we do have an ethics workshop. Is it tomorrow Trish tomorrow. So if you have questions about ethics, or whether you need brev approval to collect data from your students for your particular project, then please do join us tomorrow I believe it's 1pm so we can we can answer some of your questions there hopefully. And then lastly here what evidence should look like we have kind of a nice formula that you can try to use and practice to see if this works for your particular project. So thinking about the method the practice and then the outcome so kind of formulating a sentence based around those items. For example, you can say something like in a quality called trick survey, which is the method on the new open resource that's the practice or the output. The majority of students reported that the new resource help them prepare for the final exam. And so you're not just saying oh I designed a new open resource for the students, and they liked it, which doesn't tell us much about why they liked it or what they did with it, but framing it as it helped them prepare for the final exam. So this is the specific outcome of that project. And again this is helpful for you if you are in that final stage and you have to fill out the, the closure report and so kind of looking at some of these examples and then plugging your specific details of your project might really help you to think about whether you are kind of meeting those, meeting those evaluation criteria. Oh, thanks Alyssa. Okay I thought there was something else in the chat. Do you have anything to add to the evidence piece? I'm just going to let folks through it. Yeah. Yeah, I just wanted to add again just for those of you who are here as part of an OER fund project that yeah you'll need to provide some information about evidence on in your closure report. Even if you're not here as an official fund holder, it's really important to think about this sort of final stage as sort of providing, I'd say like closure on the project and sort of what is it that you learned and being able to form a sentence like this is really helpful in, you know, any sharing or disseminating you might want to do formally or informally, you know, being able to say to a colleague, you know, we did some focus groups from the focus groups. We were able to learn that this open website was providing students with confidence in their ability to do X, Y and Z. So just kind of thinking about that framing is really helpful. And again, you know, when you have those clear evaluation or research questions upfront, it makes it easier to sort of think about how that answer can be formulated at the end. And I just also just shared in the chat, an example of an OER project from a bunch of folks at UBC that turned into a publication. So it's just a really nice example. And it's very thorough in details, but it provides a lot of information. So I invite people to save that to look at another time. Awesome. Thank you, Trish. And our last slide is just additional resources. So if you are later on, thinking about research methods and designing surveys or questionnaires, we have a bunch of guides on our website. So if you just go to the link below, we also have tips on the BREB application process. And of course, you can always attend our workshop. We also have a BREB video, which is about 10 minutes long, but it really answers kind of some of the most common questions that we often get about BREB. But you'll find that as well on our resource page. And then of course, really important resource that we have about asking about demographics and gender. Because oftentimes we see surveys that ask a lot of kind of demographic information or questions about student sexual orientation or race or ethnicity and things like that that really have not much to do with the project. So it's really important to consider the kinds of questions that you're asking and how you're asking them in a survey and whether you need to ask them at all. So we have lots of tips and resources on our page as well on that. So please feel to check out those items. And I guess we have about eight minutes for questions now, so I will leave it open to anyone who wants to ask some questions. And Pauline, if I can jump in just one sec. So I saw a question earlier from Anita around support for evaluation support. Now I do just want to bring in another resource, since not everybody here is working on an OER fund project that there is something at UBC called the OER Fund. And these are, this is a fund to provide grants to developing and integrating open educational resources into courses. And these grants can definitely be used to help support evaluation. So you can add budget items for things like incentivizing students. So we see lots of proposals adding things like gift cards to get students to show up for focus groups or to participate in surveys. You can add grad students or assistance for helping do some of that evaluation work as well. So I'm just going to put a link into it. We're kind of off cycle. There's two grants. One grant pathway is called the implementation grants. And these are large grants for $25,000. And we're off cycle. The funding cycle for that will be in the fall. Generally, we announce them in September and the deadlines in November. But we also have rapid innovation grants. And these are called rapid because we try to get the money out the door faster. And these are small grants that are up to $2,000. And they're sort of evaluated on a monthly rolling basis. And then we can allocate proposals in at any time and generally evaluate them once per month and release funding after that. Thank you. Any questions? Jump to our last slide here where we have our emails and just invite you to also write those down. As I said earlier, in case you missed at the beginning, the session is being recorded. We're happy to share these slides with you. So you'll have access to all the links once you've moved on as well. But Pellie and I are always here. As we mentioned, we can provide some support through OER funded projects. But we're also happy just to help in as much capacity as we can with smaller questions you might have if you're really struggling with narrowing your research question or you just need a second set of eyes on a survey or some focus group questions. We're always happy to step in and spend a few minutes with you.