 Thank you very much for joining our presentation today. I think we should get started. We might have a few more people trickle in, but that's perfectly fine. So today, I'm just introducing myself. I'm Christoph Seelman. I'll be co-hosting this along with Casey Cureland, who I think you can see online as well. We're talking about hybrid courses, lessons learned from multi-campus instruction. So we're both involved, Casey and myself in multi-campus instruction. And we found that there's a lot of overlap between what we've learned there and what you might encounter in a hybrid situation. So we've put this together. So about myself, I'm in mechanical engineering. So that's a faculty of applied sciences. I started about a year and a bit ago, so I'm still quite new to UBC and rather enjoying it. Casey here is with materials engineering. So we're both involved in the manufacturing engineering program within the faculty of applied sciences, which is a joint program with UBCO, which includes shared courses. So we have courses taught at UBCV, attended by students at both campuses synchronously, and also courses taught at UBCO that are attended by students at both campuses. Synchronously, I'd also like to take this moment to acknowledge that I'm on the unceded ancestral territory of the Musqueam people, fortunately located on campus. It's absolutely gorgeous here. And I've just been loving the autumn that we've had and the brief bit of snow that we have recently as well. Very, very grateful to be here. And I'll let Casey introduce himself. Sure, yeah. Hello everyone. Good afternoon. Thanks for the good introduction so far, Christof. So I kind of describe myself as Christof's counterpart, in the materials engineering department. We both teach together and do some research projects together as well. And I'd also like to say that I'm gathered on the traditional ancestral unceded territory of the Tuassan people today, because I'm at home out here in Tuassan. So you'll be hearing from me later on. I think Christof will be leading the first bit of the session. Back to Christof. All right, great. Thank you, Casey. So today we're going to go through four areas. We'll begin with an introduction to multi-campus and hybrid instructional formats. Just, it'll be a review, I think for many of you, but just to clarify some of the, or the terminology we'll be using, I'll address that. I'm going to highlight as well here what technical abilities you need to have at your end to experience every aspect of our presentation. So listening to audio and viewing slides and contributing through Zoom chat will be part of our first module. The number two, technology and hybrid courses. So listening to audio and viewing slides will be necessary for that. For our third section, where we're going to talk about ongoing evaluation, which will be the center activity for this workshop. You'll be listening to audio speaking using a microphone and viewing slides. So please be prepared to communicate through Zoom. We're going to ask you to use a web browser and Google Docs. If you have any issue accessing these technologies and please let us know in chat, you can contact Casey directly and we can find a workaround for you. There will also be reading Zoom chat. So please have your eye on the chat when we get to that module and entering leaving Zoom breakout rooms. We will have two breakout room sessions as part of our third section in this workshop. Finally, under four, we have interpreting feedback where again, we'll just be listening to audio and viewing slides and perhaps engaging in some conversation. So I thank you very much also to those who have been posting the introductions in the chat and it's just delighted to have you here. So I appreciate you joining us. Okay, let's begin with just a very brief activity to warm things up a little bit. So we want to posit a scenario for you and you'll find this will be a theme throughout this presentation. We're going to ask you to put yourself in the mind space or headspace of students experiencing these programs through a hybrid or multi-campus format. So imagine for a moment that you're an undergraduate student at a small rural college. You're attending an engineering program. You and 15 colleagues attend a class at a remote campus. So you're in a classroom but you're watching the course through a stream. So the professor is located elsewhere with their own cohort using teleconferencing with a large university in a nearby city. Teaching assistant stands to the side watching the screen and the instructor is drawing on a whiteboard. And so everyone is focused on this screen where they're seeing this instructor right on a whiteboard through the screen. And then suddenly the screen goes blank. The TA looks startled and pulls out his phone. He's not really sure what to do. And so presumably he's contacting the professor but after minutes there's no change and the TA just looks more and more frantic as time goes on. So in Zoom chat just take a moment and think about what you might be experiencing as a student in this situation. And then under six words perhaps pose what's running through your mind or pose what emotion you would be experiencing or yeah, yeah, relate to what you're experiencing that moment in this situation. All right, excellent. Yes, we already have some coming in here. So there are so many other things I should be doing. What a waste of time. What are we missing? Anxiety, worry, waste of time. Right, so that seems to be a common theme. Will this be on the exam? Yes, excellent. I take an opportunity to have a bio break. Yes, understanding that this can happen. And we have some, yeah, a little bit of self-censorship happening as well, which is certainly reasonable considering the circumstances, especially if this is not the first time. And certainly in situations like these if technology is flaky, it may not be the first time. So we've had students in this situation and it can be a bit of a crisis to work through. And if it's a consistent issue within a course it can dramatically impact the student experience within the course and create a very big discontinuity or issue with equity between multiple cohorts. So that's one of the themes we'll be talking about today. Hi, pay good money for this. Yeah, absolutely. Okay, so we're gonna pose one more scenario. It's somewhat similar. So here you're an undergraduate student at a large university attending a course remotely from home. And so this is actually very similar perhaps to what a lot of students experienced last year in terms of attending courses, which were fully online. 100 students are attending in person. So this is a big course. Let's say we have a number of students that are in person. Let's say you're too far away from campus or it's offered in a hybrid format and so you can't come to campus every day and so you tend remotely. 50 students are attending through Zoom. Some, but not all content is available asynchronously. So we have, let's say some lesson material posted but not everything. Classes in the middle of a Q and A discussion and suddenly audio disappears and video freezes. So you're sitting at home within your quiet space perhaps your bedroom, audio disappears, video freezes. You're in the middle of a Q and A discussion and you're beginning to wonder, is this a local problem with your headset or internet? Minutes pass with no change. You don't really know potentially how to resolve the issue. So similar conversation or similar idea, what goes through your mind in this situation? Putting yourself in the headspace of students. Anxiety, frustration, panic. Once again, this question, I pay good money for this. Absolutely. But anxiety, frustration, log out, log in. Yeah, wonderful. So problem solving as well trying to fix the problem yourself. Not knowing what to do, shrugging your shoulders perhaps a bio break as well. Lost my concentration is a very, very good one. So what can happen here is it can break the rhythm of the course. If you're engaged and right in the middle of that cognitive zone where you're interested in the discussion and you're wanting to contribute and then you lose that connection that can absolutely break that sort of enchantment and put you in a position where again you're suffering anxiety and disillusionment. What am I missing? Another very important question because a lot of these Q and A discussions will lead to content that's relevant to the exams. So somehow be penalized. That's an interesting question as well. So what kind of penalties could be in place if you miss content in the course? Can't afford a better laptop, right? So accessibility is a big concern or consideration as well. So wonderful feedback and I'm glad that you're thinking in these terms because these very much I believe are questions that students have fought with over the last year or two as we've switched to online and we've had lots of new questions about how we use technology in the classroom come forth. So today there's four primary learning objectives that I'd like to cover. Discuss the importance of technology in hybrid and multi-campus education using contextual case studies contrast pedagogical technological factors that can affect equity and accessibility in multi-campus and hybrid settings. It's a very, very wordy way of saying that we want to talk about technology as pedagogy in hybrid and multi-campus courses. Apply the community of inquiry survey to assess student experience in a hybrid course. Okay, this will be our main activity for the session which will happen about halfway through and then reflect on COI survey results as part of continuous improvement in multi-campus and hybrid teaching. So how do we then use COI results to improve our delivery of our course or design of our course? I'll begin with what is multi-campus and hybrid instruction? So I've alluded to this a little bit with our introductory activities but let's begin with classical lecturing. It's something I think we're all familiar with I'm hoping or interested in at least have experienced at some point. So professor and students co-located a single location. You have a big classroom, you come in, you sit down instructors at the front behind a desk or a lectern of some sort, lecturing away. Primarily happens in the classroom very traditional in many respects. We then have remote virtual learning which is what students experienced last year certainly at UBC where the professor and students are all separated from one another. And so we have the professors sitting at home teaching to students who are all located at their own homes or perhaps small groups sharing technology. So remote virtual learning can include synchronous and asynchronous components. Next we have a flexible hybrid model or high-flex which it's a term that's I think becoming more popular it's been around for about a decade or so. This is where students choose remote in-class or purely asynchronous learning. So they have the opportunity to either work from home or work within a classroom space or attend the course asynchronously potentially with the emphasis being on flexibility. So the emphasis on student choice and learning experience they get to decide how they'd like to attend or experience a lot of the outcomes or objectives of the course. And I'll take this moment too to kind of clarify what we mean by hybrid and blended because I think those terms are often convolved or used interchangeably. So for the purposes of this presentation hybrid learning where we're treating this as an educational approach where some individuals participate in person and some participate online. And so we have a situation again where let's say we have a number of students who are located at the campus and then perhaps a variety of students who are located too far away to attend daily. They might be, this could be far up north let's say in BC or some other location where they're attending from home in a small community that's far away from their campus and therefore they attend remotely versus blended learning where instructors and facilitators combine in-person instruction with online learning activities. So this is where you're teaching in person but you also have some asynchronous content or other content online that students engage in let's say discussions for example or discussion forums. So our focus is primarily in this model here of hybrid learning. And then finally multi-campus learning. So this is multiple groups of students at separate campuses in classrooms. This is the second example that we started with. This is where we have a standard lecturing scenario classical lecturing let's say at one location but using ICT or teleconferencing. Another campus is connected remotely to the originated campus. So we have a professor at one location with one cohort and then we have one or more additional cohorts at their own campus in their own classrooms with ICT technology or communications technology that are attending the course at the same time. Oh and I should note this can happen in the same area or another part of the world. And so there are many examples of multi-campus education where they cross cultural or even language divides. So there are case studies that we know of where let's say you have students in Africa taking courses in a multi-campus format from professors in India because they lack some of that expertise in their own community. So anyway, if you're interested in multi-campus we did deliver a slideshow for last year challenges and benefits in multi-campus learning. It's available at this link which I think Casey can drop into the chat and you all can download those slides and review them. I think we also have a recording made by Ainsley as well that's available if you're interested in learning more. Okay, so without further ado I'm hoping that those terms are somewhat clear. Let's move on to talking a little bit about technology and hybrid courses. This is the second component of our presentation. So the importance of technology. Now hybrid education is very much driven by technology. Students have to connect to your stream or connect to your video or some way attend your synchronous presentation through some remote format. And unless the professor is just yelling really, really loud and there's students sitting outside you're going to have to have Zoom or some kind of teleconferencing technology provided by the university. So technology is a big part of successfully engaging in hybrid education. There's kind of two main aspects where technology plays a role. There's synchronous where the instructor engages students in real time and asynchronous where materials and activities are provided online through a learning management system like Canvas. So in a synchronous format and you have an ICT equipped room or real-time chat or face-to-face in class or Zoom or WhatsApp on your phone there's some kind of technology in place that lets the professor stream themselves through again a technology like Zoom to students who then consume or participate in the course remotely. Depending on where you are in the world this might happen through a phone. So the student might be using the phone or they'll have a computer or a laptop. It can vary. The important thing is that it's concurrent and that the instructor paces the activities. So the pacing of the class is at the behest of the instructor. They're dictating how students experience the content and they set the pacing for the content. This means a little bit less flexibility for learners but again it's a very traditional mode of teaching through synchronous. The nice thing about this is that it provides a strong sense of social and teaching presence. You're engaging in real time with the instructor and so you have that ability to interact and it gives you a stronger sense of not only teaching presence but also social engagement because you're often there engaging with colleagues as well. You're using chat. You can see colleagues around the classrooms. You have video feeds. Of course technological failure or technology failure leads to a disparity between local and remote students. If you have situations where students are attending locally and students who are attending remotely if technology fails then it's the remote students that suffer more so than the local ones. Finally, so we're next we have asynchronous enabled by learning management systems such as Canvas, video and learning videos and learning content can be posted online. Students, right? This is independently paced activity so students will consume the content at their own pace. I think we've encountered this through some of CTLT's other offerings. They have lots of courses online or workshops that require a bit of asynchronous work before you can get into the synchronous component. So for this the social engagement activities are usually short and involve some distance such as discussion boards which is great but you lack a little bit of that social engagement because it's not really as real time. So some of the technology is involved. So classrooms equipped with information communication technologies. So we have multi-campus teaching which depends on having teleconference technology inside your classroom. If you're engaging in this there are certain classrooms around UBC that are designed for teaching in this format and so they'll have microphones throughout the classroom. They'll have cameras that will follow you as you walk about and this is state-of-the-art stuff. So and that's generally what's needed to as much as possible can be that sense of presence to the remote location. So classrooms equipped with ICT technology are rare on campus but they exist and usually there's one or two per department. They're quite valuable. They require training though. They're not easy to learn how to use. Yeah, so hardware and software in this case is controlled by the university. This can be at the remote end as well where they're projecting onto a screen. Common in Okanagan campus. Yes, I've heard you guys are really going down this path which is amazing. That's fantastic. So I'm really happy to hear that. Streaming broadcasting technology such as Zoom, this is a little bit more ubiquitous. I think we're all quite familiar with this right now. It can be used in hybrid teaching. I think a lot of us have streamed our lectures over the last few months. It's a little bit less consistent in terms of classroom technology. Depends on your laptop microphone and camera depending if you're using a lapel mic or if you're using a boom mic. Some instructors perhaps are not using a special mic at all. They're just using their camera mic. So you'll see a little bit more variability in terms of quality in this sort of experience. And there's also a great deal of responsibility on behalf of the students to be prepared with the appropriate internet, computer software, and also location. And this is something that's often overlooked is that students who attend remotely have to have a quiet, comfortable space where they can attend courses remotely in a synchronous sense without a lot of distraction or other such issues. And then finally, learning management system. This is used more for asynchronous instruction which is again managed by the university. Students access it through personal or university owned hardware. So again, this is once again more under the control of the university but students still have to have the technology necessary to access the LMS. Okay, just a point from Tamara. Zoom now linked in Canvas. So hybrid between streaming and LMS. Zoom is linked in Canvas. Yeah, so we do have a bit of a mix between streaming and LMS but what's happening in Canvas is that you're basically just getting a link to Zoom. So the technology that actually that you synchronously for communications is still the Zoom infrastructure. But certainly integrating it with Canvas has made things easier. Okay, so big part of this and a very critical part when considering technology is equity and accessibility in both hybrid and multi-campus courses. This is particularly apparent in a multi-campus setting but it certainly applies in hybrid as well. So questions such as, do all students have access to the technology necessary for the course? We can't always assume that students have a laptop that lets them attend a course remotely. Do all students have a quiet, private place to experience remote course content? If students are on-site, let's say, and you're asking them to attend or on-site as an on-campus and you're asking them to attend a course remotely, do they have a place on campus they can attend such a course? That's quiet where they can talk but they can turn on their camera and they're comfortable to do so. Do they have a place at home where they can do so as well? Do they have an adequate internet connection? These are all questions of accessibility and equity and quite frankly, inclusivity as well. So we make certain assumptions about the technology and availability of resources to students who engage in these types of courses. We have to ask ourselves questions of financial hardship, noise and safe environments and also always, what if technology goes down? So if they're attending a course through their phone because that's all they can afford, what if that phone dies or what if they lose their 4G connection or something along these lines? We put a lot of oness on the students to provide the technology they need to experience courses through a hybrid setting. So it's something we should always bear in mind when preparing these courses and preparing the content because we have to respect the possibility that students may simply be unable to attend the course because of technology failure at their end. So equity of learning experience, engagement and community are three big questions that should be on your mind certainly when designing a course like this and delivering it as well. Following up on that, towards equity. So when we talk about equity, remote and local students have comparable, fulsome learning experiences. It's rather difficult to claim that it's possible for students who tend to remote course in a hybrid setting to have the same learning experience to those who are actually attending in person. I don't know if it's feasible to achieve that or even really necessary to target it. But what we can do is do our best to ensure that students attending both remotely and locally have a fulsome learning experience so that we adapt, let's say or we provide some extra provisions or consideration for students who are attending remotely that to compensate for that lack of teaching presence or to compensate possibly for technology failures. We do our best to ensure that their experience is excellent without necessarily being the same as the students that are there in person. Context is extremely important, not just the context in which the course is delivered on site but also the context that other students are experiencing either at their campus or connecting remotely through their technology. So engagement activities must be designed to encourage interaction and motivation for all participants. It's especially hard to achieve this if students are being constantly tempted to engage in other activities while attending a course remotely. So developing your activities to be heavy and interaction and motivation is very important while still adhering to learning outcomes. Three types of interaction that should be considered is student and instructor interaction but providing opportunities for remote students to engage with the instructor, student-student as well as student content. So these are three levels of interaction where activity should be constructed to all three. Okay, best practices and instructions. So some things to consider is contingencies and training. So what if a piece of technology goes down or what are my responsibilities? What are the TA student responsibilities? This is, I don't often see this well-eat out. So a lot of people who engage in hybrid instruction don't have a contingency plan in place in case technology fails. You may even want to build a worksheet in advance and meet with your TA and say, okay, if you're on site with students in a remote cohort or you notice that technology has failed. So let's say my audio has dropped out, what do you do? How do you correct the problem? How do you take over the course at their end? So having a plan in place at the beginning of the course to consider contingencies is very, very, very helpful to address sudden issues with technology. Fairness, so minimizing cohort favoritism is also quite challenging because as instructors, we feel, I think, more connected to the students with us in the room than we do to students who are connecting remotely. So overcoming our own bias and really focusing on inclusivity and engaging those students who are remote is at least personally, for me, a big challenge. And stressing Folsom learning experiences when equitable is not possible as well. To achieve all this, ongoing evaluation is critical. So this means monitoring each cohort, local students, remote students, responding quickly if problems occur and reflecting and continuously improving on your own practice. If we don't respond fast, it's amazing how fast problems can snowball out of control. If students who are attending remotely are just disengaging the course and are suffering a lot of frustration and asking questions, why did I spend money on this? As mentioned earlier, due to technical problems in the course. Oh, and an interesting contrast to this actually, just mentioned here by Tamara, heard feedback from some instructors that they get caught up focusing on those attending remotely, sometimes not paying enough attention to those in person room. Right, so finding a balance and perhaps we're all biased a little bit differently but finding a balance between the two and again, ensuring as much as possible and equitable learning experience. So just to summarize a little bit of discussion I've had right here before we move on to our next section. I have here a chart, this is provided by Balmani and Teltzfolds specifically with multi-campus instruction but doesn't apply to hybrid instruction quite a bit as well. When developing a hybrid course or multi-campus course you really should begin with contemplation, contemplation model. So programs, training and technology. What do you have to put in place before you can even begin designing your course to ensure that that course will be a success. And this goes beyond just the instructor. This also includes administration, technical resources and others. Approach it with context sensitive course design. Really consider context, both local context and remote context in designing your course and your learning activities in particular. Focus on equity and accessibility with technology in mind, engaging activities that really get students like both locally as well as remotely involved in the course and a plan for interaction. So a lot of interactivity. When engaging in delivery, think of contingencies. So what if problems occur? What problems can occur and have a plan in place so that you're not caught in the moment not really knowing what to do. Intentionally engage remote cohorts and always observe and evaluate. And then in maintenance, adapt with technology. So as technology changes make sure that you're staying on top of those changes and that the content you're delivering reflects the new opportunities that technologies or limitations that new technologies provide. Okay, and from here, I'm gonna pass it over to Casey who will take us on to evaluation. Great, thank you, Kristoff. And thanks for that great beginning to the talk today. So Kristoff's kind of laid down some of the fundamentals of the groundwork. I'll start talking about how we can evaluate that. How do you know if the course is going along well and what kind of things to look for in that regard? So first let's talk a little bit about the importance of ongoing evaluation. So with multicampus instruction then you'll have kind of two separate cohorts and they'll kind of be working together let's say socially kind of with each other usually at least, which is different from hybrid instruction where you have a cohort that's together physically together and then you'll have each individual kind of by themselves somewhat in isolation. So it's important to kind of take the temperature and see where different cohorts are at and maybe try to make sure everyone is engaged and continuing on with the course. As well the evaluation helps to inform the structure so the instructor if there's perhaps a culture of disengagement within the course hopefully we haven't experienced this but sometimes maybe one or two students start to get upset and then they kind of complain to their colleagues and then more and more kind of jumping on the bandwagon of being disengaged. Can also help you understand if there's some sort of perceived inequity. Maybe going to Tamara's comment about how the instructors focusing more on the remote students all of a sudden all the in-person students are getting a little bit upset about that and they feel like they're getting the short end of the stick. So that's something else that evaluation can help with. It can also help you identify any kind of inter-cohort animosity. So it's not uncommon where the two different cohorts will kind of become rivals against each other. Maybe one campus thinks they're different than the other campus for some reason or thinks they're better or worse or perhaps the remote students in hybrid instruction feel like they're second-rate students because they're just joining in online and then that kind of leads to some sort of animosity. It can also help understand if there's a lack of trust with the instructor. So you can get an idea of if the students just don't seem to like the instructor or don't seem to be getting along with them. And then finally opportunities for either short-term or long-term improvement maybe long-term being the next year that you teach the course. You can take the summer to improve it and deliver it next term. As well and inadequate evaluation can lead to inconsistent learning experience for the students. So perhaps the two different cohorts are getting a different experience or something along those lines. So let's go into this in a little bit more detail. So what types of evaluation are common and what do we kind of see? So self-reflection is always good in the form of post-lectured journaling. So we'll talk about this on the next slide but just kind of thinking what worked, what didn't, what could I improve on? Maybe some sort of qualitative evaluation. So either formal interviews or focus groups or just informal discussions with students maybe the five minutes before class just chat with them how are things going, what's working, what's not working or if you see them in your office hours as well. Or perhaps check in questions with the students during the lectures and make sure with these types of things it's always good to make sure that you're doing it when all the cohorts are involved. So you wouldn't really wanna do your informal discussion while the online or the remote students are not engaged at that time. And then finally quantitative types of valuations. So Kristoff and I are engineers so we like numbers and quantitative stuff. So things like course surveys that we'll talk about in more detail later on. And then the student evaluation of teaching of course which we get at the end of the term these types of more let's say concrete quantitative types of valuations. So let's go in a little bit more detail here. So self-reflective evaluation. So this is something that's self-motivated and often the form of journaling. So some sample questions to ask yourself how many students attended synchronously are the students on schedule with their homework? Are they engaged with the class keeping up with things? Did the local and remote students ask questions? Who was asking questions were the remote students kind of hogging the attention, asking all the questions because the remote students were having trouble getting word and edgewise or vice versa. Maybe the instructor is focusing more on the chat more on the remote students and not really paying attention to the locals. Did the local, sorry did they also both participate in their discussions was one more so than the other and why were there any technical problems? So hopefully not these are always or these can be a challenge to deal with but it's good to know if they can come up and good to have a contingency plan of course. As well, what did I say off camera? So typically that means kind of what did I say to the local cohort that the other cohort couldn't hear? And it might not even be something like what did I say and actually happened but what did the students that were off camera think I said or maybe think I had an interaction with the local students because that perception of having an interaction that they were missing out on can become important as well. And then of course what worked, what didn't work and what can you do better next time? Just kind of thinking about these things and trying to prepare for the next session is can be quite useful. And then let's do a little bit of a deeper dive into evaluating the student experience using what we call the community of inquiry framework and this ties into the quantitative survey that I mentioned earlier. So I'm not sure if you're familiar with this COI framework, Community of Inquiry Framework but this is kind of the diagram that we use to explain it and this is something that was developed by Garrison and his group in early 2000s and he's kind of, he's built on it since then and others have built on it and used it since then. So you can see the three main areas starting with social presence up here, cognitive presence over in green and then teaching presence down at the bottom. And you can see how they all kind of interact with each other in the Venn diagram in one way or a few ways and then they all kind of come together to create the student experience. And they all kind of fit in one way or the other. You can have maybe two that are quite strong, one that's weak and that would affect the student experience or all that are quite weak and all contribute to a poor student experience. But let's go into each of these in a bit more detail here. So starting with social presence. So this is kind of the sense of wellbeing, sorry, sense of being and belonging in a course and Garrison kind of describes it as, or defines it as being themselves socially and emotionally and kind of being a real person in the course. So for me as an instructor last term when we're online this is something that I kind of had trouble with a little bit too. Kind of thinking, I'm having trouble interacting with people on any kind of social level. It's, you always have to kind of beg them to speak up and turn their cameras on if you can even get them to turn their cameras on. But that's what we mean by social presence. We have to keep in mind too that the social interactions between the cohorts should be managed separately. So what happens kind of in one group may be different than say all the online students that are all individual. Maybe you need to make extra effort to get them to work together or get them to communicate. But yeah, and then there's also the technical challenge of course of bringing the groups together and not just between the students themselves but between the students and the instructor is important to keep in mind. Mentioned before about the potential for some sort of adversarial relationship between the two cohorts. It could potentially be healthy if it encourages students to kind of push harder and try to study harder, learn better so that they can maybe kind of compete against the other cohort hopefully in a healthy way. But if it's left then it can be very damaging and it can be a huge issue. So the next part of the COI is the cognitive presence and over here we have a figure from Garrison as well. And this kind of describes the cycle. We think of cognitive presence to kind of think of alertness or engagement being engaged with the course. So we start out with some sort of a triggering event. Often this comes from the instructor. This is kind of the point where you kind of ask the students to think about something or show them something and ask them to think a little bit about it. And then we move on to exploration where they kind of question it and brainstorm it to themselves, think about it to themselves before moving to integration where they're trying to kind of trying to construct some sort of meaning out of it. And then finally, and hopefully they get as far as this to resolve it and apply this new knowledge that they've just gained. And the idea is that if they can complete the cycle and they have a better understanding of the material. Unfortunately, activities may not resolve neatly for all students and possibly it could be the difference between one cohort is resolving all of these concepts and ideas whereas the other cohort isn't. Or maybe one cohort is only doing that with half of them or something. So we need to consider too how to keep the learners engaged when they're staring at a screen because I mean, you're all experiencing it right now. It can take a lot of motivation and effort to just keep interested, stay interested as well from the instructor should be considering that. How do you make things interesting for people? And finally, we have teaching presence. And this is perhaps where most people would think the instructor's kind of job starts and stops but the instructor does have some sort of control and influence over the other two presences as well. But Garrison defines it as kind of the binding element that creates that community. It's not just putting slides up and talking about them or delivering homework assignments. There's a lot more to it than that is as I'm sure we all know here. It kind of depends on the quality of the instructor's interactions. So our students happy to see the instructor happy to run up to them after class and ask them questions or pass by them in the hallway and smile at them or do they not wanna even see them again for the rest of the life's kind of thing. And that goes for both inside and outside of the classroom. So that can have an effect on inclusivity. Do the students that aren't physically there feel like they're part of the class or are they just kind of a fly on the wall that's observing and doing the same homework assignments? It can also have to do with equity. Are they equal or equitable to the local students? As well, does the instructor have authority in the classroom? Do people listen to them? Do they respect them? Do they feel that they're worth putting their attention towards the teacher? So we've kind of laid down the groundwork of what's in this community of inquiry. And that's all well and good but how do we actually measure these different presences and how can we try to measure this and make some sense out of it and possibly identify what's a bit lower or what's not working as well? So luckily someone else has helped us with that and done that for us. So around 2008, Arbao and all and Garrison was involved with this, but they developed a survey tool for this community of inquiry, for this COI. And it's a 34 question survey with five point Likert scale for each question. And the results are broken down into, as you could imagine, teaching presence, social presence and cognitive presence. But then within that, even more so, those three presences are broken down into further area. So within teaching presence, we have design and organization, facilitation and direct instruction. Those are all evaluated within social presence, effective expression, open communication and group cohesion. Those are all assessed and evaluated. And then finally, incognitive presence, a triggering event, exploration, integration and resolution. So the idea being that if your students take the survey, you can collect the results and kind of go over the results, see what's working for the students, what's not and then adjust, perhaps adjust your style accordingly. So something that we've been working on here at UBC is what we call the cost application, essentially community of inquiry, online survey tool, COST. And this is kind of a way to just make, using this tool easier for instructors. So it's basically putting the tool out online in a format that instructor can use. They could go into the website, create a new survey link for their course. They give that link to the students, the students fill out the survey, on their own time, maybe within a week or something. And then pretty much in real time, instantly, the tool plots the data and performs a Minova analysis on it. So that's what we can see over here on the side. And you could come to that tool, click on the button below, create a new survey for your course, generates a link, you send the link out to the students and it gives you another link that you can use to check the results from. So it's still somewhat a work in progress, but the tool is live and we'll actually see it being used later on shortly. We'll use the tool shortly after we do our activity. So that's what we'll move on over to right now. So let's do a little bit of role-playing, break up some of this talking, but let's do a role-play and we'll kind of put ourselves into the shoes of a student that's either a remote student or a local student and we'll do that by reading throughout some journal entries that they've made. So they're just three little blurbs of three different entries in a journal that they've each made and written up and we'll kind of put ourselves in their shoes, pretend that we're that student that's written those journal entries. And that'll help us highlight some of these common issues that we've been talking about. So when we get back, then we'll ask you to do the COI survey. The COI survey is that cost tool that we just discussed here. And yeah, I see your comment there, Tamara, we'll share the cost tool survey really shortly here as well. Okay, so this is how the activity is gonna work, hopefully gonna work. So our main group will be divided into four breakout rooms and I'll set those up in one second. So when you're in the room, the rooms are named after the students, these imaginary students that we've created journal entries for. So the name of your room will be the student that you're going to role play. Read through the entry and then discuss with the other people in your room. There'll be, depending on the exact numbers, maybe three, three or four people in the... Okay, thanks Christoph for posting that. Three or four people in the room with you. The link is in the chat right now, thanks Christoph. And then after the discussion, we'll move you to a second breakout room. So basically that's the breakout room of people in your cohort. So let's say you happen to be a local student, then we'll put you in a room with other local students as kind of your cohort together. Same thing with remote students, we'll put all the remote students together and just talk about your experiences, talk about kind of what sounds like what's working, what's not working from being in this class. After that, we'll bring you back to the main room and we'll share the link for the COI survey for you as well. And then when you're done the survey, if you could please just click the little zoom reactions to raise your hand or somehow indicate that you're done. And then we can move on to the next step. So let me open the rooms now. So we'll look at the results from the survey we just took very shortly. Before we do that, we just wanted to go over some results from a real course that we had. So in the unlikely event that something happens when we go to show you your results, at least we have some results here that we can look at. So before we go into the detail exactly of the results, let's talk about the class a little bit or some comments on the course. So a little backstory. So the local students had the instructor for a previous class. So they already had a little bit of a before with the instructor. They knew who the instructor was and kind of their style. The material that was used in the class with both local and remote students was taught to the local students previously in another class, but the remote students didn't learn that same material. So the local students had a little bit of an advantage in that sense or a bit of inequity, let's say, in that sense. Remote students were often distracted and kind of not paying attention if you were to walk in the classroom. You could tell they were on their phones or chatting with each other or that sort of thing. As well, the remote students complained formally a number of times about having trouble interacting with the instructor. And the remote students had trouble with the TA. The TA didn't seem to know what the instructor was teaching or what the instructor was trying to discuss or wasn't able to answer any questions really. And half the time the remote students were kind of just chatting with the TA when they should have been doing activities. So we can see the results here. So this is what will pop up for your survey as well. And this is just the first of the rest of the plot. So it starts, we can see the teaching presence, social presence and cognitive presence. And maybe a little bit small, but at the bottom we have local on the left and remote on the right. So just to begin with teaching presence right away is clear to say, to see that the local cohort, the students that were familiar with the instructor, they seem to have or perceive a much higher level for the teaching presence. And opposite is true the remote students had low perception of teaching presence there. The social presence interestingly was more or less the same. And you can kind of think of that makes a bit of sense because the groups are together with their cohort physically present with each other. So they feel that kind of social environment together perhaps not with the other cohort, but within their own cohort they would feel that. And then same thing, we see a discrepancy and cognitive presence as well. So the remote students having a bit of trouble staying focused, staying on task, staying engaged and motivated. Conversely, the local students are more cognitively present you could say. So we could look at the results on the website where they're posted and we could scroll down and then we can go into more detail on each of the subcategories within the presences. But let's not do that now. Let's look at our results and see if we can make sense of our results. So I'll pass over to Kristoff here. Okay, so let's have a look to see how we did here. So this is the data that was just prepared to access it. I went to the first link that Casey shared. So you can have a look at the data you'd like as well by accessing that link. We can see first of all at the bottom we had about five people respond from the local face-to-face and six from remote through Zoom, which is great. It's a pretty small sample set. So you'll want to get more data more people involved to get like a really good representation of what's happening inside the class. But at first blush, we can look at our candle plots right here. We have our local face-to-face on the left and our through Zoom on the right. And this is already reminding me of something I need to clean up a little bit. So on the left hand side, we have very strong teaching presence. So the in-person students felt the teacher was doing a great job. The remote students were kind of in the middle. So what we're seeing here is fairly low average, a little bit below three. Social presence, very low social presence for the remote students in this case. So they really did not feel in that they had a social contact or connection within the community, which is a very important observation because these discrepancies do highlight a lack of equity and experience. And then cognitive presence students that were in-person found the activities engaging. They really liked them. Almost a five, like almost a perfect rating on that. So the instructors doing a great job for that. But the remote students, similarly, feel a little bit less engaged. So they don't feel quite as taken into the activities. So the activities aren't quite designed or tailored to accommodate them. Since these are all quite significant. So the Minova analysis is nice. It shows us, based on the number of students, roughly where we're going to expect significance. And so the, there is a difference between the local and remote students is highly significant for teaching presence and for cognitive presence as well. And actually quite significant for social presence too. So it's very, very much within that 5% threshold which is considered significant in a quantitative analysis. So right off the bat, you can see all three of these show a very significant difference in perceived student experience between the local group and the remote group. We then do a subcategory analysis. So you can have a closer look at design organization, facilitation and direct instruction to see that difference persists throughout them. And here we see that certainly. So for the local group, there's a very, very high and pretty well universally 5.0. Everyone thought that the direct instruction was great. And the local group, the remote group, again, lukewarm didn't feel that strongly about it. Facilitation, huge difference there, big discrepancy, design organization, also very significant difference. And once again, we're seeing some statistically significant differences even though we have a very small sample size. So this is highlighting a very big problem with the organization of this course. Cognitive presence, uniform across the board too. So triggering event exploration, integration, resolution all of these are beautifully highly rated instructors doing fantastic job with the in-person students, but the remote students don't really feel it. So, and they're particularly low in something like exploration. So they don't really feel that they have the opportunity to engage with the work. They feel interested that triggering effect is there, but they don't really know where to go with it. And that might be connected to the lack of social engagement as well. So yeah, this is the results page. You're welcome to have a look at it. Again, the survey is super quick and easy to set up. It takes less than 15 seconds to get one going. You can have a look, we'll provide the link at the end of the presentation, but essentially create a new survey, name of the course, institution, your email address to get you the links, different sections, so we could have remote, local, complete. And that's it, the survey is set up. So it's super easy to set up. You then email the link to the students to fill it out. You can then look at your analysis link and get the results right there. There's no login, nothing like that. It's intended to be as easy to use as possible. And as you all witnessed, it takes about five to eight minutes to actually complete the survey. So that's the tool. Oh, I'm not sure if I'm still here, but the link is this, coimech.ubc.ca. So it's hosted by Mechanical Engineering currently. Okay, so I'm going to kick back into our slides. So thanks everyone for filling that out. Very interesting results. So you represented your students very strongly. There was good grouping and consistency between your remote students and in-person students. And a really good highlight that there were differences worthy of concern for this course. So this was a real course and the students fill this out halfway through the term. The instructor would need to act on these differences. These are highlighting big problems and how student experience is experienced in this hybrid course. Okay, so I'm just going to move through this quickly. I know we're just about out of time, but let's talk a bit about how to interpret this feedback. So things that you're going to be looking for from the COI feedback results, diverging teaching presence, which is something we saw in our case right here. So perceived inequity between instructors' attention, which means that there's possibly more focus on the local cohort or more focus on the remote cohort. In this case, the remote cohort did not feel that sense of teaching presence. So we saw that was much lower than the local group. Poor facilitation. So remote students are not able to participate in group activities. And we saw that as well cognitively, not just in teaching presence, but the remote students didn't feel engaged and even the local students noticed that. So in the journal entries, they didn't really feel connected to the remote students at all. Direct instruction, ineffective or hard to understand remote asynchronous material. If this is low and it was, then asynchronous material perhaps wasn't well understood by the remote students. Maybe they didn't get appropriate instructions or maybe the instructions were given in class off camera, which can happen too. So, and this is something we have to consider as well for a hybrid course or multi-campus course, students may actually need to be taught how to learn in this setting. They may need actual instruction, perhaps in the first lecture, on how to best experience the course and questions to ask themselves as the course progresses to ensure that they're staying on track. Trust is also very critical. And if trust is low due to a low teaching presence, then students will have even less engagement in the course. Social presence, so we have effective expression within the course. This was actually not that low. So lack of community access to peers, a limited emotional engagement and expression in course materials. So the suggestion, I think from the results that we saw from the group here was that there was still some social engagement even amongst the remote students, perhaps with one another and maybe that second meeting where the remote students could chat back and forth a little bit, helped facilitate that greater sense of social presence. But there was, I think a very strong difference between the remote students and the local students. So it was that sense of cohort or that economy or difference between the two cohorts. They weren't blending very effectively. Open communication, that was quite positive. So there wasn't a lot of difficulties with technology, which was great to see. So the instructor was conscientious about the use of technology. Nonetheless, students were not really engaging that effectively through the technology available. And then group cohesion similarly, students seemed to be forming good groups locally and less so remotely in this use case. Then lastly, cognitive, I'll be very quick here. So interest, exploration, integration, resolution. So interest, you want to be able to capture the attention of both the remote and the local students given that good trigger and temp their curiosity, exploration, varying levels of access to resources. This was particularly low for the remote students. So they clearly felt very isolated. Local students have access to libraries, they have access to study spaces, they have access to lab spaces. Remote students don't necessarily have any of that. And so this became very clear in the data that you yourselves generated, where exploration was one of the lowest rated fields under cognitive for the remote. And yet locally across the board, this was nearly perfect. So the professor was doing a great job with his in-class students. Integration, learning activity, outcomes are not consistently reaching all students. So if there's a lot of divergence here, then similarly, students are experiencing their learning outcomes differently. And resolution diverges in concept synthesis and can achieve higher learning outcomes. If the students remotely are not engaged cognitively in the work, then they're not going to achieve those high level outcomes where they can apply their knowledge. Very briefly, I'm also going to summarize a few responsibilities moving forward. So this is not delivering a successful hybrid or multi-campus course is not strictly the responsibility of the instructor. Administration has to be involved as well. So hybrid and multi-campus courses need time and resources to execute successfully. There's training required as well. TAs need to be trained if they're involved in a multi-campus course. TAs also, if they're engaging students remotely need a little bit of training as well to do so effectively, just like instructors. You also need appropriate classroom technology. So students need access to high quality microphones, cameras, laptops. If they don't have a camera on their laptop at home because it's an old laptop, they can't afford a new one then they can't engage as directly or they won't have as much presence to their peers. Classrooms also need to be appropriately equipped so that instructors have the technology they need to come across as vibrantly and realistically and brightly as possible for both local and remote students. And then of course, you need to recognize the difference between the student experience. So virtual labs versus in-person labs, student clubs, study spaces, library resources, physical presence, that context is very, very important in assessing equity, the accessibility to resources beyond just a laptop. Onto the instructor, so recognize that hybrid is not the same as face-to-face plus a video feed. And this is something that's particularly important with multi-campus and also very important for hybrid. So a very clear example of how multi-campus falls flat and fails if you just treat it like a normal course with video feed, you need to design your course from the ground up to be hybrid. You have to choose learning activities for every lesson that are more engaging and that can effectively bridge that local remote divide. So engagement is critical. And again, focus on student instructors, student-student and student-content interaction, every one of those levels. And really work to foster a healthy student community that bridges both remote and level students. Finally, be aware that as an instructor of such a course, you need to provide training for your teaching assistants and give them some indication of planning. So let them know what they can do to make the course run a little bit better. So what happens if communication fails? How do they resolve these problems? Give them a checklist they can walk through so that they're not caught unawares without a plan if things go wrong. And always keep equity and accessibility in mind through inclusive pedagogy. So that should be at the heart of your pedagogical planning. All right, and that's it. We're right on time, which is great. So good luck with your future hybrid courses. Thanks so much for attending our session. Additional resources on the following slides. So I have a few more in this presentation and also through the shared Google Doc. You'll notice at the bottom of the Google Doc there's pages of resources on hybrid instruction. I encourage you to go through those. Lots of really great stuff. Additional training if you're interested. The cost tool, you can see the link right here, coi.mech.ubc.ca. And we have a couple of, it looks like we actually have a couple of different surveys available. So this one here is one that we've put together but it looks like CTLT is asking you for one as well. So if you'd be so kind, it'll take less than two minutes to fill out our Qualtrics survey here. And it'll help us a great deal in understanding how we can improve this type of workshop in the future. So thanks so much again for coming. And I hope you have a wonderful day. Enjoy the rest of your winter institute and have a delightful holiday. We'll stick around for a couple of minutes if there are any questions. Yeah, and thanks everyone and feel free to contact us as well directly if you have questions or want to chat about something. The staff and I are both happy to engage. Yes, we're looking for collaborators. So if you're interested in this cost tool and would like to be more involved with that or try it in your class, then please let us know and we will do everything in our power to make that happen and give you the resources you need to use it. Absolutely.