 Hi everyone. Thank you for coming to our session. My name is Yan Xian. I'm a lead instructional designer from NC State University in the U.S. And I'm Steve Bader. I'm the open source associate director at NC State responsible for managing Moodle and keeping it upgraded and custom applications. Yeah, so we are from the organization called digital education and learning technology applications. We do many different things on campus, including like managing the Moodle system and also working with instructors on course design development. So we created a Moodle plugin called Roadmap that we launched across campus since last fall. Although this plugin can be used for any type of course design, our original intention was to use it to support blended and flip learning. So in this presentation, we'll talk about the background of creating this tool, working through the plugin interface, and then we'll showcase some course roadmap examples and share our findings from one of those courses. Okay, so let's get started. So blended and flip learning, which is often referred to as hybrid learning, it kind of transforms traditional in-person lecture with after-class homework into two kinds of blended flipped learning modes. So blended flipped in-person learning integrates online asynchronous learning with active learning in an in-person classroom setting, while blended flipped by-craness learning integrates online asynchronous learning with active learning in a synchronous online environment. So in both cases, students are often expected to complete online learning before they come to class for applications. So that they are ready to apply the knowledge. So at NC State, we have been using this blended flipped learning approach to redesign many of our undergraduate courses before the pandemic, and this trend has been growing even faster since the pandemic started. However, there are a number of instructional challenges with using this approach. So students are oftentimes unclear about the blended and flipped learning approach, the format, the expectations. They would show up in class without watching the lecture videos or doing the exercises, and they will just not be ready for the application, and what they can get out of the class is very limited. And also students tend to consider online learning before class as teaching myself, which they believe is not what they are paying their tuition for. So the challenge is really to help students understand the purpose of online learning and how it's preparing them for the in-class active learning. And another challenge is blended and flipped learning requires a high level of self-regulation skills to manage learning before, during, and after class, but not all students have those skills when they start college. So to address these challenges, we use this blended flipped course design framework to help organize our course materials and activities. So basically we organize the entire course into several self-regulated learning cycles in which students start by proactively planning for their learning goals, study time, learning space, study learning strategies. And then during the cycle, they frequently monitor the performance of learning the knowledge skills and in applying the strategies they plan to use. And at the end of the cycle, they kind of self-reflect on how well they achieve their goals, how adequately they use the learning strategies, and how effective the strategies work for them. And then within each self-regulated learning cycle, there can be a number of blended flipped learning cycles that are set up based on the basic learning units, such as weeks, class days, or modules. So in those cycles, students start with online self-paced learning to learn the basic concepts and skills, and then they move on to in-class active learning to apply those concepts and skills to solve problems with peer interaction and instructor guidance. And then they finally review and integrate what they have learned by solving new problems independently or with reduced support. So this framework did help us a lot with course design, but we still feel like we need a tool that can help us visually and interactively communicate this new approach and expectations to students. And that's how we came up with the idea of a model roadmap tool. And now Steve is going to show you what the tool looks like. So at the start, we went through several iterations. We looked at running with HTML and JavaScript, and then we connected it to a gamification plugin that we have in-house, and it just wasn't solving the solution. So we decided that we're going to make it a plugin. It's going to be within Moodle. It's going to display a lot like a label does. It's going to get the benefits of the Moodle APIs like user events and activity completion, and it will be able to display within the course structured and responsively so that it would look and feel more a part of Moodle. So this is kind of the display. You have your course learning objectives that are tied to the roadmap itself. They're displayed at either the top or the bottom of the roadmap. The roadmap itself consists of the bigger containers called phases. You can rename the title to whatever you'd like. The phases are color coded. They're collapsible. They turn into a progress bar not to take up space. Within phases you have cycles. Cycles have the title which you can adjust. You can name it week, day, or chapter based on whatever fits your course the best. Then within cycles you have steps. Those steps can be tied to activities in the course. It has a rollover that has a description that you can set. You can also set the expected completion date for the user so that they can see when this is supposed to be done. These steps each have like their own display, informational display to the user. The default state is not started. It's the grayed out version. But once progress is started on the activities that are associated with the step, the bar will start filling up around the circle. If the due date is approaching they'll get a little explanation point over top of it so that they can see that they're kind of in trouble. Once it's completed it'll fill in in color and if it's completed on time they get a little star. The cycles themselves allowed us to both accommodate for blended flipped learning cycles and self-regulated cycles since they're attached to activities in the course. So it could be things that they can do on their own. So the roadmap setup, we tried to make this as simple as possible since our earlier versions were very difficult. You add it to the course like any other activity. You click on the configure roadmap button and you get three sections that you have to work through. First you have to configure your course learning objectives so that they can be displayed and tied to cycles. Then you get the edit roadmap section and then you can adjust the appearance. In the edit roadmap section we wanted this to be intuitive. We display phases in a hierarchical fashion. Within the phases there are cycles and within the cycles the steps all for you to configure in one big list. They're collapsible, sortable, you can edit and you can delete all on one page as you manage. So we'll look at a closer look of those. So breaking each section down, the course learning objectives is managed by a list of text inputs and you can sort them in the list and they get a number associated to them. The phases, you can update the title, you don't necessarily have to call it a phase. It could be whatever you want to break it up in your course. The cycles within each phase, you can adjust the title, the subtitle. You can link the cycle anywhere in your course that you would like. And you will also select the course learning objectives that are tied to that cycle so that they'll display in the roadmap so the students will know what they're focusing on. Within the cycle you have steps, you can manage the rollover text. You can select from 50 of our SVG icons, some of which are on the card that are on your table. Those icons are what displays on the roadmap. You can select from activities in your course as to which ones are tied to this step. So as they complete the activities that will progress where we fill, you can link steps to places in your course. So this can act more like a menu as well as they're progressing through the course and they see where they need trouble. They can click and go right to the place in the course they need to work on. And then you can also set the expected completion date and time so that they know when this needs to be done, if it's before class. We do have the roadmap available for demo. Again, the cards that are on your table will take you to the demo website. It's also on that website there's links to the presentation which has the same links within the slide here. If you do end up on the demo server and you register, it'll send you an email for verification and whatnot. But as you register, what's going to happen is it's going to place you into a demo course as a student so that you can progress through the course as a student and see the roadmap change and show things for you. And then it'll also spin up a course for you as an instructor as a sandbox so that you can test and configure a roadmap for yourself. We do have a link to the Moodle roadmap user guide that was created in-house just in case you run into any issues. Okay, so now let's take a look at some course roadmap examples. So this is an introductory biology course. It's organized by four major learning units plus a getting started unit. So each unit has three to four weeks and every week has two to three modules. Every module has its own blended flip learning cycle with two learning steps. So conceptual learning is completed online before class to learn the basic biological concepts. And active learning happens during and after class that requires students to use the biological concepts to explain like the phenomenon they observe and also problems. We also integrated learning reflection at the beginning which is like this icon. At the beginning and then middle and end of the course which creates the self-regulated learning cycle for unit one and two together and then unit three and four together. This is a bioengineering class that has an introductory phase and then two main phases of using Excel and R to solve bioengineering problems. So each of the main phases has a number of week-long modules and each module has its own blended flip learning cycle with five learning steps. So the first two steps are completed online before class. So students first learn the tool functionalities and then they watch the instructor's problem solving demo while following along with the demo. And then they come to class in step three to solve a more complex problem with instructor guidance. And then as they move to the steps four and five they kind of work on problems more independently during lab and in homework. We also integrated learning reflection at the beginning, middle and end of the class and we made monitoring activity part of the homework problem solving. So that creates the self-regulated learning cycle for each of the main phases. So this is a game design course. So this course uses a project-based learning approach and organized the phases based on the milestones of a semester long project of designing a visual game, which is represented as the trophy icon throughout the course. So students went from understanding game design to design their game and then implement their game design and finally play test their game. So we also have the week long learning cycles, blended learning cycles. So those cycles always start with theoretical learning of the game design concepts and experiential learning by playing games. But in phase two there are two other smaller projects running concurrently as the visual game design project. So that's why there are many more steps in phase two than in phase three and four where students are focusing on implementing the game, visual game they designed and test. So this kind of shows learning cycles can be very flexible in terms of the lens. This is a Python programming course. So it's organized into four phases with increasingly complex topics. So the week long blended learning cycles has four major learning steps. So students first complete conceptual learning using lecture videos and self-check exercises before they come to class. And then during class they work on top head questions as a class, as a whole class, and then work on coding problems in smaller groups. During lab they kind of work on more complex coding problems with peer interactions and TA support. And then finally they work on homework and project activities to reinforce everything they have learned throughout the week. We have learning reflection at the beginning of the course and at the end of every phase. And we also integrated learning monitoring activities in each week. So that kind of created the self-regulated learning cycles for every single phase. So we actually did an evaluation study to look at the effectiveness of the roadmap tool in this particular course. So we collected data from 422 and Spring 23 with a total of 241 students. In both semesters we used the Moodle 4.0 with the show all sections on one page setting. So our evaluation questions included how did students use the roadmap? How did they use the roadmap use relate to their learning outcomes? And what were their perceptions and attitudes about the tool? And we collected data from student surveys, their self-reported roadmap achievements, and the course total grades. So we'll talk about our findings. So the first question we asked the students was on average how often did you look at your course roadmap? Because in Moodle 4.0 the show all sections on one page setting allows students to hide or show the section where the roadmap is located. So we believe this results kind of represent students natural tendency of looking at the roadmap. So we found three groups of students. About 15% non-users, so those are students who said they never looked at it, they never looked at the roadmap. So they probably kept the roadmap section hidden. We had about 30% of occasional users who look at the roadmap like several times in a semester or every few weeks. And we have 54 something percent frequent users who look at roadmap every week or multiple times a week. The 15% non-users was a little surprising to us because in earlier version of Moodle when the show all sections on one page setting kept the roadmap visible all the time we had less than 1% students that they never looked at it. And we also find like these three different groups of students had very different ways of using the roadmap and different perceptions attitude. So we will share the findings by comparing them. So these are the common ways students use the roadmap based on our feedback from the earlier versions of this tool. So we asked the students to rate how often they do these things. As you can see most of the non-users never really did any of these. For occasional users more than 60% of them use the roadmap for monitoring such as like check on where I am in the course track my activity completion status and use the roadmap for navigating the course. And for frequent users more than 90% students did those things frequently. For occasional users like a little less than half of them use the roadmap for planning or reflection whereas for frequent users more than 85% did those things. This is less frequent than monitor using the roadmap for monitoring which made sense to us because the roadmap tool doesn't directly support planning and reflection. But we always recommend instructors to have separate planning reflection activities to go along with using the roadmap. So these are the common perceptions about how helpful the roadmap is to the students. So again we asked the students in this class to rate how helpful they feel the tool for their learning. So as you can see like most of the non-users didn't find the tool helpful in any of these areas. More than half of the occasional users find the tool helpful and about 80 to 90% of frequent users find the tool really helpful in all these areas. We use these three statements to measure students general attitudes about the roadmap tool. So as you can see more than like frequent users are more likely to agree with all of these statements much more than the non-users and occasional users. So this kind of suggested different roadmap experience from these different groups of students. So for frequent users they tend to look at the roadmap more often and use it frequently. And they see a lot of value in using this tool and develop more positive experience and attitudes about it. Whereas for non-users and occasional users they didn't look at the roadmap or use it that often and as a result they are not seeing as much value in it. Okay and this kind of shows that okay. Yeah so let's take a look at the, we also ask students to self-report their roadmap achievements at the end of the course. We do have the plan to develop dashboards using the data captured by the roadmap tool in the future. So hopefully in the future we can get these data directly from the tool rather than relying on students self-report. So look at what we have here. So for colored icon achievements which represent the completion of activities these three different groups of students earned significantly different amount of colored icons. So with frequent users earning about 90% occasional users about 75% and non-users about 65%. So it kind of tells us that the more students look at the roadmap tool, the more they look at the roadmap, the more likely they are going to complete activities tied to the icons. And similar for the star icon achievements which represent the on-time completion of activities. The frequent users achieved significantly more star icons than the non-users or occasional users. However it still shows us that about 60% of frequent, the frequent users earn about 60% of all possible colored icons. Meaning that they will met the timeliness expectation for about 60% of the time. So we wish like this is higher percentage but this number kind of tells us two different things. One is like students like they still need to do better, they can still do better to keep up. And the other part is that this tool has some limitations that affected this number because often for example some of the activities are completed outside of Moodle. And they require students to mark it as done manually after they complete which they tend to forget to do it on time. So that affected like whether they can get to the star icon or not. And then the other part is when the activity is completed by receiving a manual grade it relies on when the instructor or the TA enter the grades in a grade book. So that also affected students achievement. So looking at the course grades distribution like a large percent of students earn the A's or B's in this class. But if you look closely at the three different groups the frequent users had higher A rates and lower B rates than the other two groups. And in this graph like these are numbers of students who earned A's in this class and among all the A's students 60% of them were frequent users. It could be because those were students that were high achievers to begin with and were more likely to meet the expectations. But as we did a regression analysis we did found that the roadmap achievements can expand about 10 to 20% of variance in the course total grades. Meaning that like the roadmap achievements could have some predictive power of the final learning outcome. So Steve we'll talk about our next steps. Yeah so we have a lot to improve on and there's some good planning going on. The first thing that we're focusing on is getting the roadmap plug into a place that we can share it with the Moodle directory. We're shooting for 4.3. We have a sign up link in our slides that you could get notified when we do finally make it available. We're working on campus to continue to add roadmap to more courses through our marketing workshops and our grant projects. And we're working to refine the current plug-in and add new features. Make it more robust and as Yann mentioned we're trying to design and develop a roadmap dashboard. And create course design recommendations related to the roadmap. Anyone have any questions for today? There you are. Hi, really interesting. Is there any reason to develop this tool as Moodle and not maybe as course format? We could go that direction but we have so many courses that are already in the current course format. And one of the beauties about the plug-in is you don't have to use it just once. You can break the roadmap plug-in into smaller pieces and add them into each section. So while a course format does fit really well and that's a great suggestion, the plug-in itself acts like a label so that you can have it in multiple places. I can see both. Thank you for sharing this presentation. It looks really fascinating. I wonder if the code is already available on GitHub or wherever you share code. And number two is have you already tested this in relation to the cause index? With what? The cause index introduced in 4.0. And the recent change that text and media activities are then sometimes displayed and the index is sometimes not so that might be an interesting thing for you. So right now it acts a lot like a label so we have that same issue where if you're editing you'll see it and if you take it away it goes away which is one of the things again mentioned was it's kind of a frustration because we lose that visibility so we're trying to figure a way to accommodate that. You don't need to do that because Moodle fixes that. I saw it was a 4.3 they address it right? Yeah, so it would be a good time for us to release and then go back to your other question. We are working towards making it a community plug-in so we would add it to the plug-in repository for Moodle and then it would be on GitHub, yes. But currently it's not the way it would be. It's not currently there. We're close. Are you willing to share it? Yeah, I can see that. Thanks for the talk. So you had mentioned one of the limitations of the roadmap is that when you have assignments that are completed outside of Moodle it is the responsibility of the students that indicate that they have completed that activity in Moodle. Do you have any thoughts on how to get around that issue? Because that could create quite a bit of difficulty if we're constantly relying on students to be diligent. It seems like it almost kind of takes the purpose of the roadmap. Yeah, so it depends on what type of activity it is outside of Moodle. Some of the activities that we have outside of Moodle but has LTI integration. So those doesn't create any problem because we can just use the external tool to connect the activity. For activities like read a certain book chapter or write some reflection notes, those kind of things. Yeah, they have to report back to Moodle to make sure they did it on time, to tell the roadmap that they did it on time. And what we found is if we are doing this consistently throughout the course, students kind of develop that routine of doing those things. So it's more like a checklist for them. I did this, I did that. So I would say for most of the students they can do that. They can follow the instructions. But of course there were always students complaining, oh I deserve that star but I couldn't get it because I forgot we always have students like that. Thank you.